Verrazzano Enterprise Container Platform – Platform Setup

Docs: Prepare an Oracle Cloud Native Environment Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Install Oracle Cloud Native Environment

Deploy Oracle Cloud Native Environment with the Kubernetes module, following instructions from Oracle Cloud Native Environment: Getting Started.

Install a Kubernetes network load balancer implementation, such as OCI-CCM or MetalLB.
Install a Container Storage Interface Driver, such as OCI-CCM.

Notes

The oci-ccm module does not elect a default StorageClass or configure policies for the CSIDrivers that it installs. A reasonable choice is the oci-bv StorageClass with its CSIDriver configured with the File group policy.
```
kubectl patch sc oci-bv -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: blockvolume.csi.oraclecloud.com
spec:
  fsGroupPolicy: File
EOF
```
Unless explicitly configured, the externalip-validation-webhook-service defaults to blocking all external IP addresses in the cluster, which causes the Verrazzano installation to fail because an IP address cannot be assigned to an ingress controller. When this situation occurs, the Verrazzano platform operator logs will contain a message similar to this:
```
admission webhook "validate-externalip.webhook.svc" denied the request: spec.externalIPs:
    Invalid value: "<external IP address>": externalIP specified is not allowed to use
```
To avoid this error, either disable the externalip-validation-webhook-service or configure the service with your load balancer IP addresses prior to installing Verrazzano. For more information, see Enabling Access to all externalIPs.

Examples

Oracle Cloud Infrastructure

The following is an example of Oracle Cloud Infrastructure that can be used to evaluate Verrazzano installed on Oracle Cloud Native Environment. If other environments are used, the capacity and configuration should be similar.

You can use the VCN Wizard of the Oracle Cloud Infrastructure Console to automatically create most of the described network infrastructure. Additional security lists and rules, as detailed in the following sections, need to be added manually. All Classless Inter-Domain Routing (CIDR) values provided are examples and can be customized as required.

Virtual Cloud Network (for example, CIDR 10.0.0.0/16)

Public Subnet for Load Balancer (for example, CIDR 10.0.0.0/24)

Security List / Ingress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type & Code	Description
No	`0.0.0.0/0`	ICMP			3, 4	ICMP errors
No	`10.0.0.0/16`	ICMP			3	ICMP errors
No	`0.0.0.0/0`	TCP	All	22		SSH
No	`0.0.0.0/0`	TCP	All	443		HTTPS load balancer

Security List / Egress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Description
No	`10.0.1.0/24`	TCP	All	22	SSH
No	`10.0.1.0/24`	TCP	All	31443	HTTPS load balancer
No	`10.0.1.0/24`	TCP	All	32443	HTTPS load balancer

Private Subnet for Kubernetes Cluster (for example, CIDR 10.0.1.0/24)

Security List / Ingress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type & Code	Description
No	`0.0.0.0/0`	ICMP			3, 4	ICMP errors
No	`10.0.0.0/16`	ICMP			3	ICMP errors
No	`10.0.0.0/16`	TCP	All	22		SSH
No	`10.0.0.0/24`	TCP	All	31443		HTTPS load balancer
No	`10.0.0.0/24`	TCP	All	32443		HTTPS load balancer
No	`10.0.1.0/24`	TCP	All	2379-2380		Kubernetes etcd
No	`10.0.1.0/24`	TCP	All	6443		Kubernetes API Server
No	`10.0.1.0/24`	TCP	All	6446		MySQL
No	`10.0.1.0/24`	TCP	All	8090-8091		Oracle Cloud Native Environment Platform Agent
No	`10.0.1.0/24`	UDP	All	8472		Flannel
No	`10.0.1.0/24`	TCP	All	10250-10255		Kubernetes Kublet

Security List / Egress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type and Code	Description
No	`10.0.0.0/16`	TCP				All egress traffic

DHCP Options

DNS Type
Internet and VCN Resolver

Route Tables

Public Subnet Route Table Rules

Destination	Target
`0.0.0.0/0`	Internet Gateway

Private Subnet Route Table Rules

Destination	Target
`0.0.0.0/0`	NAT Gateway
All Oracle Cloud Infrastructure Services	Service Gateway

Compute Instances

The following compute resources adhere to the guidelines provided in Oracle Cloud Native Environment: Getting Started. The attributes indicated (for example, Subnet, RAM, Shape, and Image) are recommendations that have been tested. Other values can be used if required.

Role	Subnet	Suggested RAM	Compatible VM Shape	Compatible VM Image
SSH Jump Host	Public	8 GB	VM.Standard3.Flex	Oracle Linux 7.9
Oracle Cloud Native Environment Operator Host	Private	16 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Control Plane Node	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Worker Node 1	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Worker Node 2	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Worker Node 3	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9

Next steps

To continue, see the Installation Guide.

Docs: Prepare an Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For an example on OKE, see Installing Calico and Setting Up Network Policies.

NOTE

OCI VCN-Native Pod Networking is now supported with Verrazzano for Kubernetes v1.26 and later. The pod network must be in a private subnet to enable egress. Either place the worker nodes in a private subnet, or place the pod network in a private subnet separate from the worker nodes. For Kubernetes v1.25 and earlier, you must use the flannel overlay network.

Create the OKE cluster using the Oracle Cloud Infrastructure Console or by some other means.
Follow the instructions provided by OKE to download the Kubernetes configuration file for your cluster, and set the following ENV variable:
```
$ export KUBECONFIG=<path to valid Kubernetes config>
```
Optional, if your organization requires the use of a private registry to the Docker images installed by Verrazzano, see Install Verrazzano in a Disconnected Environment.

Next steps

To continue, see the Installation Guide.

Docs: Configure a VCN in OCI

Mon, 01 Jan 0001 00:00:00 +0000

Before you can create Oracle Cloud Native Environment (OCNE) clusters or Oracle Container Engine for Kubernetes (OKE) clusters on Oracle Cloud Infrastructure (OCI), you’ll need to configure a virtual cloud network (VCN) in your OCI compartment. VCNs are software-defined networks that manage access to your cloud resources.

See Networking Overview in the OCI documentation for more information.

You can use the VCN Wizard in the OCI Console to automatically create most of the required network infrastructure. Additional subnets and security rules (described below) must be added manually.

Within your VCN, you’ll need:

Subnets (with security rules)
Gateways
Route tables

NOTE

In addition to the specifications listed, make sure that the VCN is configured to accept the ports and protocols required by Kubernetes. See Ports and Protocols in the Kubernetes documentation for more information.

Subnets

Subnets are subdivisions within a VCN that help to organize configuration settings. All instances within a subnet use the same route table, security lists, and DHCP options. Subnets can be either public or private. For OCNE and OKE clusters, you’ll need both public and private subnets, with four subnets in total.

See Overview of VCNs and Subnets in the OCI documentation for more information.

Each subnet requires its own set of security rules that establish rules for virtual firewalls. These ingress and egress rules specify the types of traffic (protocol and port) that are allowed in and out of the instances.

See Security Rules in the OCI documentation for more information.

NOTE

You can use either Network Security Groups (NSGs) or security lists to add security rules to your VCN. We recommend using NSGs whenever possible. For more information, see Comparison of Security Lists and Network Security Groups in the OCI documentation.

Subnet 1: control plane endpoint

A public subnet for the control plane endpoint that houses an OCI load balancer. The load balancer acts as a reverse proxy for the Kubernetes API server.

In this subnet, create security rules that cover the following traffic:

Egress: control plane traffic
Ingress: external access to the Kubernetes API endpoint
Ingress: ICMP path discovery

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.0.0/29	6443	TCP	HTTPS traffic to control plane for Kubernetes API server access

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	6443	TCP	Public access to endpoint OCI load balancer
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery

Subnet 2: control plane nodes

A private subnet that houses the control plane nodes that run Kubernetes control plane components, such as the API Server and the control plane pods.

In this subnet, create security rules that cover the following traffic:

Egress: node internet access
Ingress: east-west traffic, originating from within the VCN
Ingress: control plane endpoint to control plane node access on API endpoint
Ingress: worker nodes to control plane node access on API endpoint
Ingress: ETCD client and peer
Ingress: SSH traffic
Ingress: control plane to control plane kubelet communication
Ingress:
Ingress: Calico rules for control plane and worker nodes for
- BGP
- IP-in-IP

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	All	All	Control plane node access to the internet to pull images

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.0.8/29	6443	TCP	Kubernetes API endpoint to Kubernetes control plane communication
CIDR Block	10.0.0.0/29	6443	TCP	Control plane to control plane (API server port) communication
CIDR Block	10.0.64.0/20	6443	TCP	Worker node to Kubernetes control plane (API Server) communication
CIDR Block	10.0.0.0/29	10250	TCP	Control plane to control plane node kubelet communication
CIDR Block	10.0.0.0/29	2379	TCP	etcd client communication
CIDR Block	10.0.0.0/29	2380	TCP	etcd peer communication
CIDR Block	10.0.0.0/29	179	TCP	Calico networking (BGP)
CIDR Block	10.0.64.0/20	179	TCP	Calico networking (BGP)
CIDR Block	10.0.0.0/29		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.64.0/20		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery
CIDR Block	0.0.0.0/0	22	TCP	Inbound SSH traffic to worker nodes
CIDR Block	10.0.0.0/16	All	TCP	East-West communication for Kubernetes API server access / DNS access

Subnet 3: service load balancers

A public subnet that houses the service load balancers.

In this subnet, create security rules that cover the following traffic:

Egress: service load balancer to NodePort on worker nodes
Ingress: ICMP path discovery
Ingress: HTTP and HTTPS traffic

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.64.0/20	32000-32767	TCP	Access to NodePort services from service load balancers

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	80, 443	TCP	Incoming traffic to services
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery

Subnet 4: worker nodes

A private subnet that houses the worker nodes.

In this subnet, create security rules that cover the following traffic:

Egress: node internet access
Ingress: east-west traffic, originating from within the VCN
Ingress: SSH traffic
Ingress: ICMP path discovery
Ingress: control plane to kubelet on worker nodes
Ingress: worker node to worker node
Ingress: Calico rules for control plane and worker nodes for
- BGP
- IP-in-IP
Ingress: worker nodes to default NodePort ingress

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	All	All	Worker node access to the internet to pull images

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.0.32/27	32000-32767	TCP	Incoming traffic from service load balancers (NodePort communication)
CIDR Block	10.0.0.0/29	10250	TCP	Control plane node to worker node (kubelet communication)
CIDR Block	10.0.64.0/20	10250	TCP	Worker node to worker node (kubelet communication)
CIDR Block	10.0.0.0/29	179	TCP	Calico networking (BGP)
CIDR Block	10.0.64.0/20	179	TCP	Calico networking (BGP)
CIDR Block	10.0.0.0/29		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.64.0/20		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery
CIDR Block	0.0.0.0/0	22	22	Inbound SSH traffic to worker nodes
CIDR Block	10.0.0.0/16	All	TCP	East-West communication for Kubernetes API server access / DNS access

Gateways

Gateways control access from your VCN to other networks. You’ll need to configure three different types of gateways:

You may need to perform some additional configuration to expose the VCN’s subnets directly to the internet. See Access to the Internet in the OCI documentation for details.

Route tables

Route tables send traffic out of the VCN (for example, to the internet, to your on-premises network, or to a peered VCN) using rules that are similar to traditional network route rules.

See VCN Route Tables in the OCI documentation for more information.

For OCNE and OKE clusters, you’ll need to create two route tables:

A route table for public subnets that will route stateful traffic to and from the internet gateway. Assign this route table to both public subnets.
A route table for private subnets that will route stateful traffic to and from the NAT and service gateways. Assign this route table to both private subnets.

Docs: Prepare a Generic Kubernetes Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires that your Kubernetes cluster provides an implementation of network load balancers (Services of type LoadBalancer) for a production environment. If your generic Kubernetes implementation provides this feature, then you can use a default configuration of the Verrazzano custom resource with no customizations and follow the Installation Guide.

NOTE

Remember to not overlap network Classless Inter-Domain Routing (CIDR) blocks when designing and implementing your Kubernetes cluster; proper routing relies on that.

You can install a load balancer, such as MetalLB. This setup requires knowledge of networking both inside and outside your Kubernetes cluster. This would include specifics of your Container Network Interface (CNI) implementation, IP address allocation schemes, and routing that goes beyond the scope of this documentation. For a kind implementation, see Install and configure MetalLB.

It is possible to use a Kubernetes Service of type NodePort to test aspects of Verrazzano. This requires a good working knowledge of networking and has limited use cases.

Next steps

To continue, see the Installation Guide.

Docs: Prepare a kind Cluster

Mon, 01 Jan 0001 00:00:00 +0000

kind is a tool for running local Kubernetes clusters using Docker container “nodes”. Follow these instructions to prepare a kind cluster for running Verrazzano.

NOTE

kind is not recommended for use on macOS and Windows because the Docker network is not directly exposed to the host.

Prerequisites

Install Docker
Install kind

Prepare the kind cluster

To prepare the kind cluster for use with Verrazzano, you must create the cluster and then install and configure MetalLB in that cluster.

You can create the kind cluster in two ways: with or without image caching; image caching can speed up your installation time.

Create a kind cluster

kind images are prebuilt for each release. To find images suitable for a given release, check the release notes for your kind version (check with kind version). There you’ll find a complete listing of images created for a kind release.

The following example references a Kubernetes v1.26-based image built for kind v0.20.0. Replace that image with one suitable for the kind release you are using. For the supported Kubernetes versions, see the listing here.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.26@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

Create a kind cluster with image caching

While developing or experimenting with Verrazzano, you might destroy and re-create your kind cluster multiple times. To speed up Verrazzano installation, follow these steps to ensure that the image cache used by containerd inside a kind cluster, is preserved across clusters. Subsequent installations will be faster because they will not need to pull the images again.

1. Create a named Docker volume that will be used for the image cache and note its mountPoint path. In this example, the volume is named containerd.

$ docker volume create containerd

$ docker volume inspect containerd

# Sample output
{
    "CreatedAt": "2021-01-11T16:27:47Z",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/containerd/_data",
    "Name": "containerd",
    "Options": {},
    "Scope": "local"
}

2. Specify the mountPoint path obtained, as the hostPath under extraMounts in your kind configuration file, with a containerPath of /var/lib/containerd, which is the default containerd image caching location inside the kind container. An example of the modified kind configuration is shown in the following create cluster command.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.26@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
    extraMounts:
      - hostPath: /var/lib/docker/volumes/containerd/_data
        containerPath: /var/lib/containerd #This is the location of the image cache inside the kind container
EOF

NOTE: When using a private container registry or experiencing rate-limiting of container image pulls, refer to the kind documentation.

Install and configure MetalLB

By default, kind does not provide an implementation of network load balancers (Services of type LoadBalancer). MetalLB offers a network load balancer implementation.

To install MetalLB:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml --wait=true

Wait for MetalLB to be ready, as shown:

$ kubectl get pods -n metallb-system -L app=metallb
NAME                         READY   STATUS    RESTARTS       AGE   APP=METALLB
controller-d9d8c78b6-hhplg   1/1     Running   1 (3d4h ago)   9d    
speaker-4lqpg                1/1     Running   1 (3d4h ago)   9d

For further details, see the MetalLB installation guide.

MetalLB is idle until configured. Configure MetalLB in Layer 2 mode and give it control over a range of IP addresses in the kind Docker network. In versions v0.7.0 and earlier, kind uses Docker’s default bridge network; in versions v0.8.0 and later, it creates its own bridge network in kind.

To determine the subnet of the kind Docker network in kind v0.8.0 and later:

$ docker inspect kind | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.18.0.0/16

To determine the subnet of the kind Docker network in kind v0.7.0 and earlier:

$ docker inspect bridge | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.17.0.0/16

For use by MetalLB, assign a range of IP addresses at the end of the kind network’s subnet CIDR range. To set an address range, use the ADDRESS_RANGE environment variable:

ADDRESS_RANGE="172.18.0.230-172.18.0.254"

#Create the IPAddressPool for the cluster

$ kubectl apply -f - <<-EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: vzlocalpool
  namespace: metallb-system
spec:
  addresses:
  - ${ADDRESS_RANGE}
EOF

#Create the L2Advertisment resource for the cluster

$ kubectl apply -f - <<-EOF
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: vzmetallb
  namespace: metallb-system
spec:
  ipAddressPools:
  - vzlocalpool
EOF

Next steps

To continue, see the Installation Guide.