Verrazzano Enterprise Container Platform – Provision Additional Clusters

Docs: Cluster API

Mon, 01 Jan 0001 00:00:00 +0000

The Cluster API (CAPI) project was developed as a sub-project of Kubernetes and strives to standardize a set of Kubernetes-style APIs for cluster management. External organizations can then build upon these standard APIs to develop custom cluster management solutions.

Learn more about CAPI at Kubernetes Cluster API Documentation.

Verrazzano incorporates CAPI functionality through the clusterAPI component, which provides the ability to quickly design and deploy clusters and then continue managing your clusters throughout their life cycle, all from within Verrazzano.

NOTE

Verrazzano and Cluster API use slightly different terminology for the same concepts:

Admin cluster (Verrazzano) = management cluster (Cluster API)
Managed cluster (Verrazzano) = workload cluster (Cluster API)

CAPI splits cluster management responsibilities across three main components, which it calls providers:

Infrastructure providers standardize the host environment by provisioning any infrastructure or computational resources required by the cluster or machine.
Bootstrap providers streamline the node creation process by converting servers into Kubernetes nodes.
Control plane providers work with the Kubernetes API to regulate your clusters, ensuring that they always strive toward a desired state.

Currently, Verrazzano supports the CAPI provider for Oracle Cloud Native Environment (CAPOCNE) which bundles a bootstrap and a control plane provider together and works with the CAPOCI infrastructure provider offered by Oracle Cloud Infrastructure (OCI).

During the setup process, the bootstrap provider converts a cluster into an admin cluster - a Kubernetes cluster that controls any other, subordinate or managed clusters. It generates certificates, starts and manages the creation of additional nodes, and handles the addition of control plane and worker nodes to the cluster.

Next, a CAPI infrastructure provider will provision the first instance on the cloud provider and generate a provider ID, a unique identifier that any future nodes and clusters will use to associate with the instance. It will also create a kubeconfig file. The first control plane node is ready after these are created.

After the admin cluster is up and running, you can use the clusterAPI component to create additional managed clusters.

NOTE

The Cluster API provides its own command line tool, clusterctl, to manage the lifecycle operations of a cluster API admin cluster. Do not use clusterctl to manage any OCNE or OKE clusters on OCI that you created in the console. You will create conflicts between changes made in the console and changes made with clusterctl.

Docs: Create OCNE Clusters Running on OCI

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

You’ll need to:

Set up an Oracle Cloud Infrastructure (OCI) account with
- A compartment
- An API signing key
Generate an SSH key pair to use for cluster authentication

Create a new OCNE cluster on OCI

To provision new Oracle Cloud Native Environment (OCNE) managed clusters on OCI, complete the following steps:

Log in to the console. To find the console URL for your cluster, refer to Get console URLs and use the rancherURL value.
Open the navigation menu and select Cluster Management.
From the left menu, select Cloud Credentials, and then click Create. Cloud credentials store the credentials for your cloud infrastructure provider.
Choose Oracle.
Provide a name for the cloud credential and then fill in the rest of the fields with information from your OCI account and its API signing key.
- fingerprint: The fingerprint of the public API key. Find your key’s fingerprint.
- passphrase: The passphrase used for the API key, if it was encrypted.
- privateKeyContents: Copy the contents of the private key portion of the API key pair. Generate an API Signing Key.
- region: Enter the identifier for the current region of your tenancy. Find your region identifier.
- tenancyId: Enter the OCID of your tenancy. Find your tenancy OCID.
- userId: Enter the OCID of the user. Find your user OCID.
Click Create to save the new cloud credential.
From the left menu, select Clusters, and then click Create.
Select Oracle OCNE on OCI and provide a name for the cluster.
Expand Member Roles to add any users that you want grant access to this cluster and their permissions.
Expand Labels and Annotations to configure Kubernetes labels and annotations for the cluster.
Select the cloud credentials that you created. Ensure that the appropriate Region and Compartment are selected from their drop-down menus.
Click Next.
Set up your network. Choose Quick Create to create a new virtual cloud network (VCN) configured to the specifications required for an OCNE cluster or Existing Infrastructure to use a VCN that’s already configured in your OCI account.
- If you choose the Existing Infrastructure option, then select the compartment where your VCN is located from the VCN Compartment drop-down menu, then the VCN itself from the Virtual Cloud Network drop-down menu. Next, select subnets within the VCN for each of the Cloud Plane Subnet, Load Balancer Subnet, and Worker Node Subnet drop-down menus. See Configure a VCN in OCI for requirements.
- The VCN compartment does not need to match the compartment specified in the cloud credential.
Click Next.
Choose a Node Image from the drop-down menu.

You can choose to override the default Node Image. Expand OCNE Image Configuration and provide a custom image OCID. If your custom image includes OCNE binaries, then you can select Skip OCNE Dependency Installation to avoid duplicates.
Copy or upload an SSH public key to manage authentication of the cluster. Your SSH public key is installed on the cluster nodes, enabling SSH after the cluster is created.
Configure the cluster control plane. Select the OCNE Version first because it determines which Kubernetes versions are available, then choose a Kubernetes Version and a Control Plane Shape. You can leave the rest of the options at their default setting or modify them as needed.

Under Advanced, you can choose to edit image tags for ETCD, CoreDNS, and Calico, or whether to install OCI CCM/CSI and Calico.
Add node pools to your cluster. Clusters without node pools will schedule pods on control plane nodes.
(Optional) Install Verrazzano on the cluster. Choose a Verrazzano version from the drop-down menu. You can also expand Advanced to make changes to the Verrazzano Resource YAML. By default, Verrazzano is installed using the managed-cluster profile which enables a limited set of components on the cluster.
Expand Advanced Settings to make additional modifications to the default settings of your new cluster.
- YAML Manifests: Supply additional YAML manifests that are automatically installed after cluster creation. The total size of all additional YAML manifests may not exceed 500 KB.
- Cluster Networking: Configure cluster IP ranges and proxy settings.
- Container Registry: Specify a private registry for your container.
Click Create. It can take up to 30 minutes to provision all of the resources for your cluster, particularly for multi-node clusters.

When your cluster finishes provisioning, you can access it from the main Cluster Management page.

For help troubleshooting cluster creation, see Cluster Creation Issues.

Docs: Create OKE Clusters Running on OCI

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

You’ll need to:

Set up an Oracle Cloud Infrastructure (OCI) account with
- A compartment
- An API signing key
Generate an SSH key pair to use for cluster authentication

Create a new OKE cluster on OCI

To provision new Oracle Container Engine for Kubernetes (OKE) managed clusters on OCI, complete the following steps:

Log in to the console. To find the console URL for your cluster, refer to Get console URLs and use the rancherURL value.
Open the navigation menu and select Cluster Management.
From the left menu, select Cloud Credentials, and then click Create. Cloud credentials store the credentials for your cloud infrastructure provider.
Choose Oracle.
Provide a name for the cloud credential and then fill in the rest of the fields with information from your OCI account and its API signing key.
- fingerprint: The fingerprint of the public API key. Find your key’s fingerprint.
- passphrase: The passphrase used for the API key, if it was encrypted.
- privateKeyContents: Copy the contents of the private key portion of the API key pair. Generate an API Signing Key.
- region: Enter the identifier for the current region of your tenancy. Find your region identifier.
- tenancyId: Enter the OCID of your tenancy. Find your tenancy OCID.
- userId: Enter the OCID of the user. Find your user OCID.
From the left menu, select Clusters, and then click Create.
Select Oracle OKE and provide a name for the cluster. Do not select Oracle OKE (Legacy).
Expand Member Roles to add any users that you want grant access to this cluster and their permissions.
Expand Labels and Annotations to configure Kubernetes labels and annotations for the cluster.
Select the cloud credentials that you created. Ensure that the appropriate Region and Compartment are selected from their drop-down menus.
Click Next.
Set up your network. Choose Quick Create to create a new virtual cloud network (VCN) configured to the specifications required for an OKE cluster or Existing Infrastructure to use a VCN that’s already configured in your OCI account.
- If you choose the Existing Infrastructure option, then select the compartment where your VCN is located from the VCN Compartment drop-down menu, then the VCN itself from the Virtual Cloud Network drop-down menu. Next, select subnets within the VCN for each of the Cloud Plane Subnet, Load Balancer Subnet, and Worker Node Subnet drop-down menus. See Configure a VCN in OCI for requirements.
- The VCN compartment does not need to match the compartment specified in the cloud credential.
Click Next.
Configure the cluster control plane. Select an OKE Version and then a CNI Type.
Choose a Node Image from the drop-down menu.
Copy or upload an SSH public key to manage authentication of the cluster. Your SSH public key is installed on the cluster nodes, enabling SSH after the cluster is created.
Add node pools to your cluster. Clusters without node pools will schedule pods on control plane nodes.
(Optional) Install Verrazzano on the cluster. Choose a Verrazzano version from the drop-down menu. You can also expand Advanced to make changes to the Verrazzano Resource YAML. By default, Verrazzano is installed using the managed-cluster profile which enables a limited set of components on the cluster.
Expand Advanced Settings to make additional modifications to the default settings of your new cluster.
- YAML Manifests: Supply additional YAML manifests that are automatically installed after cluster creation. The total size of all additional YAML manifests may not exceed 500 KB.
- Cluster Networking: Configure cluster IP ranges and proxy settings.
- Container Registry: Specify a private registry for your container.
Click Create. It can take up to 30 minutes to provision all of the resources for your cluster, particularly for multi-node clusters.

When your cluster finishes provisioning, you can access it from the main Cluster Management page.

For help troubleshooting cluster creation, see Cluster Creation Issues.

Docs: Customize clusterAPI

Mon, 01 Jan 0001 00:00:00 +0000

The clusterAPI component allows you to quickly create managed clusters and manage them in the Verrazzano console. See Cluster API for more information.

You can customize the clusterAPI component using component overrides in the Verrazzano custom resource. Refer to the clusterAPI reference to see which overrides are available.

Upgrade providers

You can upgrade the individual providers that make up the clusterAPI component. This allows you to take advantage of new features in the providers without upgrading your entire Verrazzano installation.

This example customizes the clusterAPI component as follows:

Sets the version of both the OCNE bootstrap provider and the OCNE control plane provider to 1.6.1

NOTE: Because the OCNE bootstrap and OCNE control plane providers are bundled together, make sure to set their overrides to the same version or it may lead to unexpected behavior.
Sets the version of the OCI infrastructure provider to 0.9.0

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
   clusterAPI:
       enabled: true
       overrides:
       - values:
           defaultProviders:
               ocneBootstrap:
                   version: 1.6.1
               ocneControlPlane:
                   version: 1.6.1
               oci:
                   version: v0.9.0

Use a private registry

If you want to upgrade the clusterAPI providers but your Verrazzano instance is installed in a disconnected environment, you can configure the clusterAPI component to retrieve the provider assets from another location, instead of the public repository.

Place the provider assets in a location that is accessible by your disconnected Verrazzano environment.
For each provider that you want to upgrade, add a url override and then enter the path to the provider assets in the private registry for your environment.

For example:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
   clusterAPI:
       enabled: true
       overrides:
       - values:
           defaultProviders:
               ocneBootstrap:
                   url: https://my.private.network/cluster-api-provider-ocne/releases/tag/v1.6.1
               ocneControlPlane:
                   url: https://my.private.network/cluster-api-provider-ocne/releases/tag/v1.6.1
               oci:
                   url: https://my.private.network/cluster-api-provider-oci/releases/tag/v0.9.0

Docs: Configure Verrazzano on Managed Clusters

Mon, 01 Jan 0001 00:00:00 +0000

When you provision new clusters using the cluster API component, you can also choose to install Verrazzano on them. Then, you can use VerrazzanoFleet, a cluster API add-on, to perform lifecycle management operations on Verrazzano installations on your managed clusters.

Configure VerrazzanoFleet

Before you can manage Verrazzano on your managed clusters, you need to configure the VerrazzanoFleet resource.

Provision OCNE or OKE clusters. For instructions, see:
- Create OCNE Clusters Running on OCI
- Create OKE Clusters Running on OCI
Identify the namespace where the new cluster resides.
```
$ kubectl get clusters.cluster.x-k8s.io -A
```
Create a VerrazzanoFleet resource. The following example creates a typical VerrazzanoFleet resource.
```
$ kubectl apply -f - <<EOF
  apiVersion: addons.cluster.x-k8s.io/v1alpha1
  kind: VerrazzanoFleet
  metadata:
    name: example-fleet-1
    namespace: default
  spec:
    clusterSelector:
      name: cluster1
    verrazzano:
      spec:
        profile: managed-cluster
EOF
```

Customize a remote Verrazzano installation

By default, new managed clusters are provisioned with the Verrazzano managed-cluster profile. If you want to enable more components, disable unnecessary ones, or modify the settings of a component, then you can edit the verrazzano.spec object of the VerrazzanoFleet resource to suit your needs.

Use the same process to edit the verrazzano.spec object as you would the Verrazzano resource. For more information, see Modify Verrazzano Installations.

To see which components are already enabled in a managed-cluster profile, refer to Profile Configurations.

The following configuration shows you how to enable Argo CD and Velero.

$ kubectl apply -f - <<EOF
  apiVersion: addons.cluster.x-k8s.io/v1alpha1
  kind: VerrazzanoFleet
  metadata:
    name: example-fleet-1
    namespace: default
  spec:
    clusterSelector:
      name: cluster1
    verrazzano:
      spec:
        profile: managed-cluster
        components:
          argoCD:
            enabled: true
          velero: 
            enabled: true
EOF

Upgrade Verrazzano on managed clusters

You can upgrade the Verrazzano installation on managed clusters.

Upgrade the Verrazzano installation on the admin cluster. For instructions, see Upgrade Verrazzano.
In the VerrazzanoFleet resource, use component overrides to upgrade VerrazzanoFleet.spec.verrazzano.spec.version. The Verrazzano version on the managed clusters must match the version of the admin cluster.

In this example, the managed clusters are upgraded to Verrazzano 1.7.0.
```
$ kubectl apply -f - <<EOF
  apiVersion: addons.cluster.x-k8s.io/v1alpha1
  kind: VerrazzanoFleet
  metadata:
    name: example-fleet-1
    namespace: default
  spec:
    clusterSelector:
      name: cluster1 
    verrazzano:
      spec:
        profile: managed-cluster
        version: 1.7.0
EOF
```

Use a private registry

You can manage your managed clusters even when they are installed in a disconnected environment. Use component overrides to configure the VerrazzanoFleet resource to access a private registry.

To connect to a private registry, you need to configure the following component overrides:

VerrazzanoFleet.spec.imagePullSecrets.name
VerrazzanoFleet.spec.image.repository
VerrazzanoFleet.spec.image.tag
VerrazzanoFleet.spec.privateRegistry.enabled

For example:

$ kubectl apply -f - <<EOF
  apiVersion: addons.cluster.x-k8s.io/v1alpha1
  kind: VerrazzanoFleet
  metadata:
    name: example-fleet-1
    namespace: default
  spec:
    clusterSelector:
      name: cluster1
    imagePullSecrets:
    - name: verrazzano-container-registry
    image:
      repository: ${OCNE_IMAGE_REPOSITORY=ghcr.io}/${VZ_IMAGE_PATH=verrazzano}
      tag: ${VERRAZZANO_PLATFORM_OPERATOR_IMAGE_TAG}
    privateRegistry:
       enabled: true
    verrazzano:
      spec:
        profile: managed-cluster
EOF

Remove Verrazzano from managed clusters

You can remove the Verrazzano installation from managed clusters.

On the admin cluster, run this command:

$ kubectl delete vf -n <namespace> <name of verrazzanofleet>

When a VerrazzanoFleet object is deleted, it removes Verrazzano from the cluster which was associated with the fleet.