Verrazzano Enterprise Container Platform – Advanced Features

Docs: Create OCNE Clusters Running on Microsoft Azure

Mon, 01 Jan 0001 00:00:00 +0000

NOTE

This feature is experimental. It has not been thoroughly tested and is provided for informational purposes only. We make no guarantees about its safety or stability and do not recommend implementing this feature in production environments.

The Cluster API project provides a standard set of Kubernetes-style APIs for cluster management. Officially, Verrazzano currently only supports using Cluster API to provision OCNE and OKE clusters on OCI.

However, you can also experiment with using the features of the Cluster API project directly to deploy OCNE clusters on Microsoft Azure.

For more information on Cluster API or Cluster API with Azure, see:

NOTE

Verrazzano and Cluster API use slightly different terminology for the same concepts:

Admin cluster (Verrazzano) = management cluster (Cluster API)
Managed cluster (Verrazzano) = workload cluster (Cluster API)

Prepare Azure resources

Before you can deploy a Cluster API cluster, you need to set up a few resources in Azure.

Install the Azure command-line interface (CLI) tool. For instructions, see How to install the Azure CLI in the Microsoft Azure documentation.
Create an Azure resource group. In the Azure CLI, run the following command:
```
$ az group create --name <ResourceGroupName> --location <location>
```
For more detailed instructions, see Manage Azure Resource Groups by using Azure CLI in the Microsoft Azure documentation.
Create a service principal. Make sure it has the privileges it needs to create resources. This means a contributor role at minimum. The following example creates a service principal, assigns it the contributor role, and defines its scope.
```
$ az ad sp create-for-rbac  --name myServicePrincipalName \
                            --role Contributor \
                            --scopes /subscriptions/mySubscriptionID/resourceGroups/myResourceGroupName
```
For more detailed instructions, see Create an Azure service principal with Azure CLI in the Microsoft Azure documentation.

Set up the admin cluster

The Cluster API requires an initial cluster as a starting point to deploy its resources.

Install kind. Follow the instructions at Installation in the kind documentation.
Create a Kubernetes cluster using kind. Follow the instructions at Quick Start: Install and/or configure a Kubernetes cluster in The Cluster API Book.
Install the clusterctl CLI tool. clusterctl manages the lifecycle operations of a cluster API admin cluster. Follow the instructions at Quick Start: Install clusterctl in the Cluster API Book.
Install the Verrazzano CLI tool using the instructions at CLI Setup.
Install Verrazzano on the cluster using either the dev or prod installation profile. Follow the instructions at Install with CLI. The certManager and clusterAPI components are required and must remain enabled.
On the cluster, set environment variables for the following Azure resource IDs from your Azure account and from the service principal you created:
- Subscription ID
- Tenant ID
- Client ID
- Client Secret
For example:
```
# Azure resource  IDs
$ export AZURE_SUBSCRIPTION_ID="<SubscriptionId>"
$ export AZURE_TENANT_ID="<Tenant>"
$ export AZURE_CLIENT_ID="<AppId>"
$ export AZURE_CLIENT_SECRET="<Password>"

# Base64 encode the Azure Resource IDs
$ export AZURE_SUBSCRIPTION_ID_B64="$(echo -n "$AZURE_SUBSCRIPTION_ID" | base64 | tr -d '\n')"
$ export AZURE_TENANT_ID_B64="$(echo -n "$AZURE_TENANT_ID" | base64 | tr -d '\n')"
$ export AZURE_CLIENT_ID_B64="$(echo -n "$AZURE_CLIENT_ID" | base64 | tr -d '\n')"
$ export AZURE_CLIENT_SECRET_B64="$(echo -n "$AZURE_CLIENT_SECRET" | base64 | tr -d '\n')"

# Settings needed for AzureClusterIdentity used by the AzureCluster
$ export AZURE_CLUSTER_IDENTITY_SECRET_NAME="<cluster-identity-secret>"
$ export CLUSTER_IDENTITY_NAME="<cluster-identity>"
$ export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default"
```
Create a secret that includes the password of the service principal identity created in Azure. This secret is referenced by the AzureClusterIdentity used by the AzureCluster.
```
$ kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}"
```
Install the Cluster API Azure infrastructure provider.
```
$ clusterctl init -n verrazzano-capi -i azure
```
clusterctl will report when the admin cluster was successfully initialized.

Create a managed cluster

The Cluster API uses a cluster template to deploy a predefined set of Cluster API objects and create a managed cluster.

Set the following environment variables so they are available to the cluster template. Update the values to reflect your own environment.
```
# Base64 encoded SSH key for node access
$ export AZURE_SSH_PUBLIC_KEY_B64="<sshKey>"
  
# Select VM types.
$ export AZURE_CONTROL_PLANE_MACHINE_TYPE="Standard_D2s_v3"
$ export AZURE_NODE_MACHINE_TYPE="Standard_D2s_v3"
 
# [Optional] Select resource group. The default value is ${CLUSTER_NAME}.
$ export AZURE_RESOURCE_GROUP="<resourceGroupName>
 
# Name of the Azure datacenter location. Change this value to your desired location.
$ export AZURE_LOCATION="<location>"
 
# Cluster name info
$ export CLUSTER_NAME="capi-quickstart"
$ export KUBERNETES_VERSION="<k8sVersion>"
$ export NAMESPACE="default"
$ export CONTROL_PLANE_MACHINE_COUNT="1"
$ export WORKER_MACHINE_COUNT="1"
```

Copy the cluster template and save it locally as azure-capi.yaml.

Click here for the cluster template

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: OCNEControlPlane
    name: ${CLUSTER_NAME}-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AzureCluster
    name: ${CLUSTER_NAME}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureCluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: default
spec:
  identityRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AzureClusterIdentity
    name: ${CLUSTER_IDENTITY_NAME}
  location: ${AZURE_LOCATION}
  networkSpec:
    subnets:
      - name: control-plane-subnet
        role: control-plane
      - name: node-subnet
        role: node
    vnet:
      name: ${AZURE_VNET_NAME:=${CLUSTER_NAME}-vnet}
  resourceGroup: ${AZURE_RESOURCE_GROUP:=${CLUSTER_NAME}}
  subscriptionID: ${AZURE_SUBSCRIPTION_ID}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: default
spec:
  template:
    spec:
      image:
        marketplace:
          publisher: "Oracle"
          offer: "Oracle-Linux"
          sku: "ol88-lvm-gen2"
          version: "8.8.3"
      dataDisks:
        - diskSizeGB: 256
          lun: 0
          nameSuffix: etcddisk
      osDisk:
        diskSizeGB: 128
        osType: Linux
      sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""}
      vmSize: ${AZURE_CONTROL_PLANE_MACHINE_TYPE}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: default
spec:
  clusterName: ${CLUSTER_NAME}
  replicas: ${WORKER_MACHINE_COUNT}
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: OCNEConfigTemplate
          name: ${CLUSTER_NAME}-md-0
      clusterName: ${CLUSTER_NAME}
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: AzureMachineTemplate
        name: ${CLUSTER_NAME}-md-0
      version: ${KUBERNETES_VERSION}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: default
spec:
  template:
    spec:
      image:
        marketplace:
          publisher: "Oracle"
          offer: "Oracle-Linux"
          sku: "ol88-lvm-gen2"
          version: "8.8.3"
      osDisk:
        diskSizeGB: 128
        osType: Linux
      sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""}
      vmSize: ${AZURE_NODE_MACHINE_TYPE}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureClusterIdentity
metadata:
  labels:
    clusterctl.cluster.x-k8s.io/move-hierarchy: "true"
  name: ${CLUSTER_IDENTITY_NAME}
  namespace: default
spec:
  allowedNamespaces: {}
  clientID: ${AZURE_CLIENT_ID}
  clientSecret:
    name: ${AZURE_CLUSTER_IDENTITY_SECRET_NAME}
    namespace: ${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}
  tenantID: ${AZURE_TENANT_ID}
  type: ServicePrincipal
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: OCNEControlPlane
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: default
spec:
  moduleOperator:
    enabled: true
  verrazzanoPlatformOperator:
    enabled: true
  controlPlaneConfig:
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: external
        certSANs:
          - localhost
          - 127.0.0.1
      dns:
        imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
        imageTag: ${DNS_TAG=v1.9.3}
      etcd:
        local:
          imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
          imageTag: ${ETCD_TAG=3.5.6}
      controllerManager:
        extraArgs:
          cloud-provider: external
      networking: {}
      scheduler: {}
      imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
    files:
      - contentFrom:
          secret:
            key: control-plane-azure.json
            name: ${CLUSTER_NAME}-control-plane-azure-json
        owner: root:root
        path: /etc/kubernetes/azure.json
        permissions: "0644"
    initConfiguration:
      nodeRegistration:
        criSocket: /var/run/crio/crio.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ local_hostname }}'
    joinConfiguration:
      discovery: {}
      nodeRegistration:
        criSocket: /var/run/crio/crio.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ local_hostname }}'
    preOCNECommands:
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6"
        >/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }} {{ local_hostname }} localhost
        localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts
    users:
      - name: opc
        sudo: ALL=(ALL) NOPASSWD:ALL
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: AzureMachineTemplate
      name: ${CLUSTER_NAME}-control-plane
      namespace: default
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: ${KUBERNETES_VERSION}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: OCNEConfigTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: default
spec:
  template:
    spec:
      clusterConfiguration:
        imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            cloud-provider: external
          name: '{{ local_hostname }}'
      preOCNECommands:
        - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
        - echo "::1         ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6"
          >/etc/hosts
        - echo "127.0.0.1   {{ ds.meta_data.hostname }} {{ local_hostname }} localhost
          localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts
      users:
        - name: opc
          sudo: ALL=(ALL) NOPASSWD:ALL

Generate and apply the template by running the following command:
```
$ clusterctl generate yaml --from azure-capi.yaml | kubectl apply -f -
```

To view the status of the cluster and its resources, run:

$ clusterctl describe cluster $CLUSTER_NAME

To get the kubeconfig file, run:

$ clusterctl get kubeconfig ${CLUSTER_NAME} > ${CLUSTER_NAME}.kubeconfig

Finish cluster configuration

After the cluster resources are created, you must perform some additional steps to finish the configuration of the cluster.

Install a cloud controller manager (CCM). A CCM is necessary when deploying cloud resources such as load balancers.
```
$ helm install --kubeconfig=./${CLUSTER_NAME}.kubeconfig --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set infra.clusterName=clusterName --set cloudControllerManager.clusterCIDR="192.168.0.0/16" --set cloudControllerManager.caCertDir=/etc/pki/ca-trust
```
Install a container network interface (CNI). The following example uses the Calico CNI.
```
$ helm repo add projectcalico https://docs.tigera.io/calico/charts --kubeconfig=./${CLUSTER_NAME}.kubeconfig && \
$ helm install calico projectcalico/tigera-operator --kubeconfig=./${CLUSTER_NAME}.kubeconfig -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico/values.yaml --namespace tigera-operator --create-namespace
```

Your admin cluster and first managed cluster are now up and running and ready to deploy applications. You can add more managed clusters as needed.

For more information, refer to the documentation for Cluster API and Cluster API Azure:

Troubleshoot the deployment

If the deployment of the Azure resources fails, then you can check the following log files to diagnose the issue:

The Azure cluster controller provider logs:

$ kubectl logs -n verrazzano-capi -l cluster.x-k8s.io/provider=infrastructure-azure

The OCNE control plane provider logs:

$ kubectl logs -n verrazzano-capi -l cluster.x-k8s.io/provider=control-plane-ocne

NOTE: If a pod enters a CrashLoopBackOff state, then you can either restart the deployment or wait for the state to run its course. This is a known issue that should not affect the deployment of your cluster.

Delete the clusters

Delete the managed clusters.
```
$ kubectl delete cluster $CLUSTER_NAME
```
Delete the admin cluster.
```
$ kind delete cluster
```

Do not use kubectl delete -f capi-quickstart.yaml to delete the entire cluster template at once because it might leave behind pending resources that you need to clean up manually.

Docs: Create OCNE Clusters Running on VMware vSphere

Mon, 01 Jan 0001 00:00:00 +0000

NOTE

However, you can also experiment with using the features of the Cluster API project directly to deploy OCNE clusters on VMware vSphere.

For more information on Cluster API or Cluster API with vSphere, see:

NOTE

Verrazzano and Cluster API use slightly different terminology for the same concepts:

Admin cluster (Verrazzano) = management cluster (Cluster API)
Managed cluster (Verrazzano) = workload cluster (Cluster API)

Before you begin

If you have an existing vSphere environment, you can ignore Set up a VMware Software-Defined Data Center and start from Prepare the VM environment. Confirm that your environment meets the requirements as specified at Cluster API Provider vSphere: Install Requirements.

Otherwise, create a vSphere environment. We recommend using the Oracle Cloud VMware Solution as described in Set up a VMware Software-Defined Data Center. It deploys a VMware software-defined data center (SDDC) on Oracle Cloud Infrastructure (OCI) and then integrates it with other Oracle services running on Oracle Cloud. This solution was developed in partnership with VMware to provide an environment that adheres to best practices recommended by VMware.

For more information on the Oracle Cloud VMware Solution, see Deploy a highly available VMware-based SDDC to the cloud in the Oracle Help Architecture Center.

Set up a VMware Software-Defined Data Center

NOTE

Skip this section if you have configured a vSphere environment or a VMware SDDC already.

Use the Oracle Cloud VMware Solution to rapidly create a VMware SDDC.

Set up a virtual cloud network (VCN). You can choose to use an existing VCN or let the Oracle Cloud VMware Solution create its own VCN as part of the SDDC provisioning process. If you use an existing VCN, then make sure it meets the requirements defined in Prepare Your Deployment in the Oracle Help Architecture Center.
Deploy the SDDC. To request a new VMware SDDC on OCI, follow the instructions at Deploy the SDDC in the Oracle Help Architecture Center.
Ensure that the various components were created successfully. Follow the instructions at Monitor the SDDC Creation Process in the Oracle Help Architecture Center.

Prepare the VM environment

Download an Oracle Linux 8 ISO image from Oracle Linux Installation Media.
Upload the Oracle Linux 8 ISO image to vSphere. Use the steps at Upload ISO Image Installation Media for a Guest Operating System in the vSphere documentation.
Deploy a VM by following the instructions at Create a Virtual Machine with the New Virtual Machine Wizard in the vSphere documentation.
Install cloud-init on the VM.
```
$ sudo yum install -y cloud-init
```
Initialize cloud-init.
```
$ cloud-init init --local
```
When cloud-init is successfully configured, it returns a message similar to the following:
```
$ cloud-init v. 20.1.0011 running 'init-local' at Fri, 01 Apr 2022 01:26:11 +0000. Up 38.70 seconds.
```
Shut down the VM.
Convert the VM into a template and name it OL8-Base-Template. Follow the instructions at Clone a Virtual Machine to a Template in the vSphere documentation.

Set up the admin cluster

The Cluster API requires an initial cluster as a starting point to deploy its resources.

Install kind. Follow the instructions at Installation in the kind documentation.
Create a Kubernetes cluster using kind. This cluster must be accessible by the VMware SDDC. Follow the instructions at Quick Start: Install and/or configure a Kubernetes cluster in The Cluster API Book.
Install the clusterctl CLI tool. clusterctl manages the lifecycle operations of a cluster API admin cluster. Follow instructions at Quick Start: Install clusterctl in the Cluster API Book.
Install the Verrazzano CLI tool using the instructions at CLI Setup.
Install Verrazzano on the cluster using either the dev or prod profile. Follow the instructions at Install with CLI.
On the cluster, set the following vSphere environment variables. Update the values to reflect your own environment.
```
$ export VSPHERE_PASSWORD="<vmware-password>"
$ export VSPHERE_USERNAME="administrator@vsphere.local"
$ export VSPHERE_SERVER="<IP address or FQDN>"
$ export VSPHERE_DATACENTER="<SDDC-Datacenter>"
$ export VSPHERE_DATASTORE="<vSAN-Datastore>"	
$ export VSPHERE_NETWORK="workload"
$ export VSPHERE_RESOURCE_POOL="*/Resources/Workload"
$ export VSPHERE_FOLDER="<folder-name>"
$ export VSPHERE_TEMPLATE="OL8-Base-Template"
$ export VSPHERE_SSH_AUTHORIZED_KEY="<Public-SSH-Authorized-Key>"
$ export VSPHERE_TLS_THUMBPRINT="<SHA1 thumbprint of vCenter certificate>"
$ export VSPHERE_STORAGE_POLICY=""
$ export CONTROL_PLANE_ENDPOINT_IP="<IP address or FQDN>"
```
For information on the values of the environment variables, see Configuring and installing Cluster API Provider vSphere in a management cluster in the Cluster API Provider vSphere documentation.
Install the Cluster API Provider vSphere to initialize the admin cluster.
```
$ clusterctl init -n verrazzano-capi -i vsphere
```
clusterctl will report when the admin cluster was successfully initialized.

Create a managed cluster

The Cluster API uses a cluster template to deploy a predefined set of Cluster API objects and create a managed cluster.

Copy the cluster template and save it locally as vsphere-capi.yaml.

Click here for the cluster template

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - ${POD_CIDR=192.168.0.0/16}
    serviceDomain: cluster.local
    services:
      cidrBlocks:
        - ${CLUSTER_CIDR=10.128.0.0/12}
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: OCNEControlPlane
    name: ${CLUSTER_NAME}-control-plane
    namespace: ${NAMESPACE}
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereCluster
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereCluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  controlPlaneEndpoint:
    host: ${CONTROL_PLANE_ENDPOINT_IP}
    port: 6443
  identityRef:
    kind: Secret
    name: ${CLUSTER_NAME}
  server: ${VSPHERE_SERVER}
  thumbprint: '${VSPHERE_TLS_THUMBPRINT}'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: ${NAMESPACE}
spec:
  template:
    spec:
      cloneMode: linkedClone
      datacenter: ${VSPHERE_DATACENTER=oci-w01dc}
      datastore: ${VSPHERE_DATASTORE=vsanDatastore}
      diskGiB: ${VSPHERE_DISK=200}
      folder: ${VSPHERE_FOLDER=CAPI}
      memoryMiB: ${VSPHERE_MEMORY=32384}
      network:
        devices:
          - dhcp4: true
            networkName: "${VSPHERE_NETWORK=workload}"
      numCPUs: ${VSPHERE_CPU=4}
      os: Linux
      resourcePool: '${VSPHERE_RESOURCE_POOL=*/Resources/Workload}'
      server: '${VSPHERE_SERVER=11.0.11.130}'
      storagePolicyName: ${VSPHERE_STORAGE_POLICY=""}
      template: ${VSPHERE_TEMPLATE=OL8-Base-Template}
      thumbprint: '${VSPHERE_TLS_THUMBPRINT}'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: ${NAMESPACE}
spec:
  template:
    spec:
      cloneMode: linkedClone
      datacenter: ${VSPHERE_DATACENTER=oci-w01dc}
      datastore: ${VSPHERE_DATASTORE=vsanDatastore}
      diskGiB: ${VSPHERE_DISK=200}
      folder: ${VSPHERE_FOLDER=CAPI}
      memoryMiB: ${VSPHERE_MEMORY=32384}
      network:
        devices:
          - dhcp4: true
            networkName: "${VSPHERE_NETWORK=workload}"
      numCPUs: ${VSPHERE_CPU=4}
      os: Linux
      resourcePool: '${VSPHERE_RESOURCE_POOL=*/Resources/Workload}'
      server: '${VSPHERE_SERVER=11.0.11.130}'
      storagePolicyName: ${VSPHERE_STORAGE_POLICY=""}
      template: ${VSPHERE_TEMPLATE=OL8-Base-Template}
      thumbprint: '${VSPHERE_TLS_THUMBPRINT}'
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: OCNEControlPlane
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: ${NAMESPACE}
spec:
  moduleOperator:
    enabled: true
  verrazzanoPlatformOperator:
    enabled: true
  controlPlaneConfig:
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: external
        certSANs:
          - localhost
          - 127.0.0.1
      dns:
        imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
        imageTag: ${DNS_TAG=v1.9.3}
      etcd:
        local:
          imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
          imageTag: ${ETCD_TAG=3.5.6}
      controllerManager:
        extraArgs:
          cloud-provider: external
      networking: {}
      scheduler: {}
      imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
    files:
      - content: |
          apiVersion: v1
          kind: Pod
          metadata: 
            creationTimestamp: null
            name: kube-vip
            namespace: kube-system
          spec: 
            containers: 
            - args: 
              - manager
              env: 
              - name: cp_enable
                value: "true"
              - name: vip_interface
                value: ""
              - name: address
                value: ${CONTROL_PLANE_ENDPOINT_IP}
              - name: port
                value: "6443"
              - name: vip_arp
                value: "true"
              - name: vip_leaderelection
                value: "true"
              - name: vip_leaseduration
                value: "15"
              - name: vip_renewdeadline
                value: "10"
              - name: vip_retryperiod
                value: "2"
              image: ghcr.io/kube-vip/kube-vip:v0.5.11
              imagePullPolicy: IfNotPresent
              name: kube-vip
              resources: {}
              securityContext: 
                capabilities: 
                  add: 
                  - NET_ADMIN
                  - NET_RAW
              volumeMounts: 
              - mountPath: /etc/kubernetes/admin.conf
                name: kubeconfig
            hostAliases: 
            - hostnames: 
              - kubernetes
              ip: 127.0.0.1
            hostNetwork: true
            volumes: 
            - hostPath: 
                path: /etc/kubernetes/admin.conf
                type: FileOrCreate
              name: kubeconfig
          status: {}
        owner: root:root
        path: /etc/kubernetes/manifests/kube-vip.yaml
    initConfiguration:
      nodeRegistration:
        criSocket: /var/run/crio/crio.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ local_hostname }}'
    joinConfiguration:
      discovery: {}
      nodeRegistration:
        criSocket: /var/run/crio/crio.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ local_hostname }}'
    verbosity: 9
    preOCNECommands:
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6"
        >/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }} {{ local_hostname }} localhost
        localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts
    users:
      - name: opc
        sshAuthorizedKeys:
          - ${VSPHERE_SSH_AUTHORIZED_KEY}
        sudo: ALL=(ALL) NOPASSWD:ALL
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereMachineTemplate
      name: ${CLUSTER_NAME}-control-plane
      namespace: ${NAMESPACE}
  replicas: ${CONTROL_PLANE_MACHINE_COUNT=1}
  version: ${KUBERNETES_VERSION=v1.26.6}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: OCNEConfigTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: ${NAMESPACE}
spec:
  template:
    spec:
      clusterConfiguration:
        imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne}
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            cloud-provider: external
          name: '{{ local_hostname }}'
      verbosity: 9
      preOCNECommands:
        - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
        - echo "::1         ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6"
          >/etc/hosts
        - echo "127.0.0.1   {{ ds.meta_data.hostname }} {{ local_hostname }} localhost
          localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts
      users:
        - name: opc
          sshAuthorizedKeys:
            - ${VSPHERE_SSH_AUTHORIZED_KEY}
          sudo: ALL=(ALL) NOPASSWD:ALL
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-md-0
  namespace: ${NAMESPACE}
spec:
  clusterName: ${CLUSTER_NAME}
  replicas: ${NODE_MACHINE_COUNT=3}
  selector:
    matchLabels: {}
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
          kind: OCNEConfigTemplate
          name: ${CLUSTER_NAME}-md-0
      clusterName: ${CLUSTER_NAME}
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: VSphereMachineTemplate
        name: ${CLUSTER_NAME}-md-0
      version: ${KUBERNETES_VERSION=v1.26.6}
---
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSet
metadata:
  name: ${CLUSTER_NAME}-crs-0
  namespace: ${NAMESPACE}
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  resources:
    - kind: Secret
      name: ${CLUSTER_NAME}-vsphere-csi-controller
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-controller-role
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-controller-binding
    - kind: Secret
      name: ${CLUSTER_NAME}-csi-vsphere-config
    - kind: ConfigMap
      name: csi.vsphere.vmware.com
    - kind: ConfigMap
      name: vsphere-csi-controller-sa
    - kind: ConfigMap
      name: vsphere-csi-node-sa
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role-binding
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-node-role
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-node-binding
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-internal-feature-states.csi.vsphere.vmware.com
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-controller-service
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-controller
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-node
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-vsphere-csi-node-windows
    - kind: Secret
      name: ${CLUSTER_NAME}-cloud-controller-manager
    - kind: Secret
      name: ${CLUSTER_NAME}-cloud-provider-vsphere-credentials
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-cpi-manifests
  strategy: Reconcile
---
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
stringData:
  password: ${VSPHERE_PASSWORD}
  username: ${VSPHERE_USERNAME}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-controller
  namespace: ${NAMESPACE}
data:
  data: |
    apiVersion: v1
    kind: ServiceAccount
    metadata: 
      name: vsphere-csi-controller
      namespace: kube-system
---
apiVersion: v1
data:
  data: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata: 
      name: vsphere-csi-controller-role
    rules: 
    - apiGroups: [""]
      resources: ["nodes", "pods", "configmaps"]
      verbs: ["get", "list", "watch"]
    - apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
    - apiGroups: [""]
      resources: ["persistentvolumeclaims/status"]
      verbs: ["patch"]
    - apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
    - apiGroups: [""]
      resources: ["events"]
      verbs: ["get", "list", "watch", "create", "update", "patch"]
    - apiGroups: ["coordination.k8s.io"]
      resources: ["leases"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    - apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses", "csinodes"]
      verbs: ["get", "list", "watch"]
    - apiGroups: ["storage.k8s.io"]
      resources: ["volumeattachments"]
      verbs: ["get", "list", "watch", "patch"]
    - apiGroups: ["cns.vmware.com"]
      resources: ["triggercsifullsyncs"]
      verbs: ["create", "get", "update", "watch", "list"]
    - apiGroups: ["cns.vmware.com"]
      resources: ["cnsvspherevolumemigrations"]
      verbs: ["create", "get", "list", "watch", "update", "delete"]
    - apiGroups: ["apiextensions.k8s.io"]
      resources: ["customresourcedefinitions"]
      verbs: ["get", "create", "update"]
    - apiGroups: ["storage.k8s.io"]
      resources: ["volumeattachments/status"]
      verbs: ["patch"]
    - apiGroups: ["cns.vmware.com"]
      resources: ["cnsvolumeoperationrequests"]
      verbs: ["create", "get", "list", "update", "delete"]
    - apiGroups: [ "snapshot.storage.k8s.io" ]
      resources: [ "volumesnapshots" ]
      verbs: [ "get", "list" ]
    - apiGroups: [ "snapshot.storage.k8s.io" ]
      resources: [ "volumesnapshotclasses" ]
      verbs: [ "watch", "get", "list" ]
    - apiGroups: [ "snapshot.storage.k8s.io" ]
      resources: [ "volumesnapshotcontents" ]
      verbs: [ "create", "get", "list", "watch", "update", "delete", "patch"]
    - apiGroups: [ "snapshot.storage.k8s.io" ]
      resources: [ "volumesnapshotcontents/status" ]
      verbs: [ "update", "patch" ]
    - apiGroups: [ "cns.vmware.com" ]
      resources: [ "csinodetopologies" ]
      verbs: ["get", "update", "watch", "list"]
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-controller-role
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata: 
      name: vsphere-csi-controller-binding
    roleRef: 
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: vsphere-csi-controller-role
    subjects: 
    - kind: ServiceAccount
      name: vsphere-csi-controller
      namespace: kube-system
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-controller-binding
  namespace: ${NAMESPACE}
---
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}-csi-vsphere-config
  namespace: ${NAMESPACE}
stringData:
  data: |
    apiVersion: v1
    kind: Secret
    metadata: 
      name: csi-vsphere-config
      namespace: kube-system
    stringData: 
      csi-vsphere.conf: |+
        [Global]
        thumbprint = "${VSPHERE_TLS_THUMBPRINT}"
        cluster-id = "${NAMESPACE}/${CLUSTER_NAME}"

        [VirtualCenter "${VSPHERE_SERVER}"]
        insecure-flag = "true"
        user = "${VSPHERE_USERNAME}"
        password = "${VSPHERE_PASSWORD}"
        datacenters = "${VSPHERE_DATACENTER}"
        targetvSANFileShareDatastoreURLs = "${VSPHERE_DATASTORE_URL_SAN}"

        [Network]
        public-network = "${VSPHERE_NETWORK=workload}"

    type: Opaque
type: addons.cluster.x-k8s.io/resource-set
---
apiVersion: v1
data:
  data: |
    apiVersion: storage.k8s.io/v1
    kind: CSIDriver
    metadata:
      name: csi.vsphere.vmware.com
    spec:
      attachRequired: true
      podInfoOnMount: false
kind: ConfigMap
metadata:
  name: csi.vsphere.vmware.com
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: ServiceAccount
    apiVersion: v1
    metadata:
      name: vsphere-csi-controller
      namespace: kube-system
kind: ConfigMap
metadata:
  name: vsphere-csi-controller-sa
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: ServiceAccount
    apiVersion: v1
    metadata:
      name: vsphere-csi-node
      namespace: kube-system
kind: ConfigMap
metadata:
  name: vsphere-csi-node-sa
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata: 
      name: vsphere-csi-node-cluster-role
    rules: 
      - apiGroups: ["cns.vmware.com"]
        resources: ["csinodetopologies"]
        verbs: ["create", "watch", "get", "patch"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get"]
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: vsphere-csi-node-cluster-role-binding
    subjects:
      - kind: ServiceAccount
        name: vsphere-csi-node
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: vsphere-csi-node-cluster-role
      apiGroup: rbac.authorization.k8s.io
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role-binding
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: vsphere-csi-node-role
      namespace: kube-system
    rules:
      - apiGroups: [""]
        resources: ["configmaps"]
        verbs: ["get", "list", "watch"]
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-node-role
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: vsphere-csi-node-binding
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: vsphere-csi-node
        namespace: kube-system
    roleRef:
      kind: Role
      name: vsphere-csi-node-role
      apiGroup: rbac.authorization.k8s.io
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-node-binding
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    apiVersion: v1
    data:
      "csi-migration": "true"
      "csi-auth-check": "true"
      "online-volume-extend": "true"
      "trigger-csi-fullsync": "false"
      "async-query-volume": "true"
      "improved-csi-idempotency": "true"
      "improved-volume-topology": "true"
      "block-volume-snapshot": "true"
      "csi-windows-support": "false"
      "use-csinode-id": "true"
      "list-volumes": "false"
      "pv-to-backingdiskobjectid-mapping": "false"
      "cnsmgr-suspend-create-volume": "true"
      "topology-preferential-datastores": "true"
      "max-pvscsi-targets-per-vm": "true"
    kind: ConfigMap
    metadata:
      name: internal-feature-states.csi.vsphere.vmware.com
      namespace: kube-system
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-internal-feature-states.csi.vsphere.vmware.com
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    apiVersion: v1
    kind: Service
    metadata:
      name: vsphere-csi-controller
      namespace: kube-system
      labels:
        app: vsphere-csi-controller
    spec:
      ports:
        - name: ctlr
          port: 2112
          targetPort: 2112
          protocol: TCP
        - name: syncer
          port: 2113
          targetPort: 2113
          protocol: TCP
      selector:
        app: vsphere-csi-controller
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-controller-service
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: vsphere-csi-controller
      namespace: kube-system
    spec:
      replicas: 1
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
          maxSurge: 0
      selector:
        matchLabels:
          app: vsphere-csi-controller
      template:
        metadata:
          labels:
            app: vsphere-csi-controller
            role: vsphere-csi
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - vsphere-csi-controller
                  topologyKey: "kubernetes.io/hostname"
          serviceAccountName: vsphere-csi-controller
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
            - key: node-role.kubernetes.io/master
              operator: Exists
              effect: NoSchedule
            - key: node-role.kubernetes.io/control-plane
              operator: Exists
              effect: NoSchedule
            # uncomment below toleration if you need an aggressive pod eviction in case when
            # node becomes not-ready or unreachable. Default is 300 seconds if not specified.
            #- key: node.kubernetes.io/not-ready
            #  operator: Exists
            #  effect: NoExecute
            #  tolerationSeconds: 30
            #- key: node.kubernetes.io/unreachable
            #  operator: Exists
            #  effect: NoExecute
            #  tolerationSeconds: 30
          dnsPolicy: "Default"
          containers:
            - name: csi-attacher
              image: k8s.gcr.io/sig-storage/csi-attacher:v3.5.0
              args:
                - "--v=4"
                - "--timeout=300s"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
                - "--kube-api-qps=100"
                - "--kube-api-burst=100"
              env:
                - name: ADDRESS
                  value: /csi/csi.sock
              volumeMounts:
                - mountPath: /csi
                  name: socket-dir
            - name: csi-resizer
              image: k8s.gcr.io/sig-storage/csi-resizer:v1.5.0
              args:
                - "--v=4"
                - "--timeout=300s"
                - "--handle-volume-inuse-error=false"
                - "--csi-address=$(ADDRESS)"
                - "--kube-api-qps=100"
                - "--kube-api-burst=100"
                - "--leader-election"
              env:
                - name: ADDRESS
                  value: /csi/csi.sock
              volumeMounts:
                - mountPath: /csi
                  name: socket-dir
            - name: vsphere-csi-controller
              image: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0
              args:
                - "--fss-name=internal-feature-states.csi.vsphere.vmware.com"
                - "--fss-namespace=$(CSI_NAMESPACE)"
              imagePullPolicy: "Always"
              env:
                - name: CSI_ENDPOINT
                  value: unix:///csi/csi.sock
                - name: X_CSI_MODE
                  value: "controller"
                - name: X_CSI_SPEC_DISABLE_LEN_CHECK
                  value: "true"
                - name: X_CSI_SERIAL_VOL_ACCESS_TIMEOUT
                  value: 3m
                - name: VSPHERE_CSI_CONFIG
                  value: "/etc/cloud/csi-vsphere.conf"
                - name: LOGGER_LEVEL
                  value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION
                - name: INCLUSTER_CLIENT_QPS
                  value: "100"
                - name: INCLUSTER_CLIENT_BURST
                  value: "100"
                - name: CSI_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              volumeMounts:
                - mountPath: /etc/cloud
                  name: vsphere-config-volume
                  readOnly: true
                - mountPath: /csi
                  name: socket-dir
              ports:
                - name: healthz
                  containerPort: 9808
                  protocol: TCP
                - name: prometheus
                  containerPort: 2112
                  protocol: TCP
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: healthz
                initialDelaySeconds: 10
                timeoutSeconds: 3
                periodSeconds: 5
                failureThreshold: 3
            - name: liveness-probe
              image: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0
              args:
                - "--v=4"
                - "--csi-address=/csi/csi.sock"
              volumeMounts:
                - name: socket-dir
                  mountPath: /csi
            - name: vsphere-syncer
              image: gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.7.0
              args:
                - "--leader-election"
                - "--fss-name=internal-feature-states.csi.vsphere.vmware.com"
                - "--fss-namespace=$(CSI_NAMESPACE)"
              imagePullPolicy: "Always"
              ports:
                - containerPort: 2113
                  name: prometheus
                  protocol: TCP
              env:
                - name: FULL_SYNC_INTERVAL_MINUTES
                  value: "30"
                - name: VSPHERE_CSI_CONFIG
                  value: "/etc/cloud/csi-vsphere.conf"
                - name: LOGGER_LEVEL
                  value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION
                - name: INCLUSTER_CLIENT_QPS
                  value: "100"
                - name: INCLUSTER_CLIENT_BURST
                  value: "100"
                - name: GODEBUG
                  value: x509sha1=1
                - name: CSI_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              volumeMounts:
                - mountPath: /etc/cloud
                  name: vsphere-config-volume
                  readOnly: true
            - name: csi-provisioner
              image: k8s.gcr.io/sig-storage/csi-provisioner:v3.2.1
              args:
                - "--v=4"
                - "--timeout=300s"
                - "--csi-address=$(ADDRESS)"
                - "--kube-api-qps=100"
                - "--kube-api-burst=100"
                - "--leader-election"
                - "--default-fstype=ext4"
                # needed only for topology aware setup
                #- "--feature-gates=Topology=true"
                #- "--strict-topology"
              env:
                - name: ADDRESS
                  value: /csi/csi.sock
              volumeMounts:
                - mountPath: /csi
                  name: socket-dir
            - name: csi-snapshotter
              image: k8s.gcr.io/sig-storage/csi-snapshotter:v6.0.1
              args:
                - "--v=4"
                - "--kube-api-qps=100"
                - "--kube-api-burst=100"
                - "--timeout=300s"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
              env:
                - name: ADDRESS
                  value: /csi/csi.sock
              volumeMounts:
                - mountPath: /csi
                  name: socket-dir
          volumes:
            - name: vsphere-config-volume
              secret:
                secretName: csi-vsphere-config
            - name: socket-dir
              emptyDir: {}

kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-controller
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: vsphere-csi-node
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: vsphere-csi-node
      updateStrategy:
        type: "RollingUpdate"
        rollingUpdate:
          maxUnavailable: 1
      template:
        metadata:
          labels:
            app: vsphere-csi-node
            role: vsphere-csi
        spec:
          nodeSelector:
            kubernetes.io/os: linux
          serviceAccountName: vsphere-csi-node
          hostNetwork: true
          dnsPolicy: "ClusterFirstWithHostNet"
          containers:
            - name: node-driver-registrar
              image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
              env:
                - name: ADDRESS
                  value: /csi/csi.sock
                - name: DRIVER_REG_SOCK_PATH
                  value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
              volumeMounts:
                - name: plugin-dir
                  mountPath: /csi
                - name: registration-dir
                  mountPath: /registration
              livenessProbe:
                exec:
                  command:
                    - /csi-node-driver-registrar
                    - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
                    - --mode=kubelet-registration-probe
                initialDelaySeconds: 3
            - name: vsphere-csi-node
              image: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0
              args:
                - "--fss-name=internal-feature-states.csi.vsphere.vmware.com"
                - "--fss-namespace=$(CSI_NAMESPACE)"
              imagePullPolicy: "Always"
              env:
                - name: NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: CSI_ENDPOINT
                  value: unix:///csi/csi.sock
                - name: MAX_VOLUMES_PER_NODE
                  value: "59" # Maximum number of volumes that controller can publish to the node. If value is not set or zero Kubernetes decide how many volumes can be published by the controller to the node.
                - name: X_CSI_MODE
                  value: "node"
                - name: X_CSI_SPEC_REQ_VALIDATION
                  value: "false"
                - name: X_CSI_SPEC_DISABLE_LEN_CHECK
                  value: "true"
                - name: LOGGER_LEVEL
                  value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION
                - name: GODEBUG
                  value: x509sha1=1
                - name: CSI_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: NODEGETINFO_WATCH_TIMEOUT_MINUTES
                  value: "1"
              securityContext:
                privileged: true
                capabilities:
                  add: ["SYS_ADMIN"]
                allowPrivilegeEscalation: true
              volumeMounts:
                - name: plugin-dir
                  mountPath: /csi
                - name: pods-mount-dir
                  mountPath: /var/lib/kubelet
                  # needed so that any mounts setup inside this container are
                  # propagated back to the host machine.
                  mountPropagation: "Bidirectional"
                - name: device-dir
                  mountPath: /dev
                - name: blocks-dir
                  mountPath: /sys/block
                - name: sys-devices-dir
                  mountPath: /sys/devices
              ports:
                - name: healthz
                  containerPort: 9808
                  protocol: TCP
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: healthz
                initialDelaySeconds: 10
                timeoutSeconds: 5
                periodSeconds: 5
                failureThreshold: 3
            - name: liveness-probe
              image: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0
              args:
                - "--v=4"
                - "--csi-address=/csi/csi.sock"
              volumeMounts:
                - name: plugin-dir
                  mountPath: /csi
          volumes:
            - name: registration-dir
              hostPath:
                path: /var/lib/kubelet/plugins_registry
                type: Directory
            - name: plugin-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
                type: DirectoryOrCreate
            - name: pods-mount-dir
              hostPath:
                path: /var/lib/kubelet
                type: Directory
            - name: device-dir
              hostPath:
                path: /dev
            - name: blocks-dir
              hostPath:
                path: /sys/block
                type: Directory
            - name: sys-devices-dir
              hostPath:
                path: /sys/devices
                type: Directory
          tolerations:
            - effect: NoExecute
              operator: Exists
            - effect: NoSchedule
              operator: Exists
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-node
  namespace: ${NAMESPACE}
---
apiVersion: v1
data:
  data: |
    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: vsphere-csi-node-windows
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: vsphere-csi-node-windows
      updateStrategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
      template:
        metadata:
          labels:
            app: vsphere-csi-node-windows
            role: vsphere-csi-windows
        spec:
          nodeSelector:
            kubernetes.io/os: windows
          serviceAccountName: vsphere-csi-node
          containers:
            - name: node-driver-registrar
              image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
              env:
                - name: ADDRESS
                  value: 'unix://C:\\csi\\csi.sock'
                - name: DRIVER_REG_SOCK_PATH
                  value: 'C:\\var\\lib\\kubelet\\plugins\\csi.vsphere.vmware.com\\csi.sock'
              volumeMounts:
                - name: plugin-dir
                  mountPath: /csi
                - name: registration-dir
                  mountPath: /registration
              livenessProbe:
                exec:
                  command:
                    - /csi-node-driver-registrar.exe
                    - --kubelet-registration-path=C:\\var\\lib\\kubelet\\plugins\\csi.vsphere.vmware.com\\csi.sock
                    - --mode=kubelet-registration-probe
                initialDelaySeconds: 3
            - name: vsphere-csi-node
              image: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0
              args:
                - "--fss-name=internal-feature-states.csi.vsphere.vmware.com"
                - "--fss-namespace=$(CSI_NAMESPACE)"
              imagePullPolicy: "Always"
              env:
                - name: NODE_NAME
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: spec.nodeName
                - name: CSI_ENDPOINT
                  value: 'unix://C:\\csi\\csi.sock'
                - name: MAX_VOLUMES_PER_NODE
                  value: "59" # Maximum number of volumes that controller can publish to the node. If value is not set or zero Kubernetes decide how many volumes can be published by the controller to the node.
                - name: X_CSI_MODE
                  value: node
                - name: X_CSI_SPEC_REQ_VALIDATION
                  value: 'false'
                - name: X_CSI_SPEC_DISABLE_LEN_CHECK
                  value: "true"
                - name: LOGGER_LEVEL
                  value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION
                - name: X_CSI_LOG_LEVEL
                  value: DEBUG
                - name: CSI_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: NODEGETINFO_WATCH_TIMEOUT_MINUTES
                  value: "1"
              volumeMounts:
                - name: plugin-dir
                  mountPath: 'C:\csi'
                - name: pods-mount-dir
                  mountPath: 'C:\var\lib\kubelet'
                - name: csi-proxy-volume-v1
                  mountPath: \\.\pipe\csi-proxy-volume-v1
                - name: csi-proxy-filesystem-v1
                  mountPath: \\.\pipe\csi-proxy-filesystem-v1
                - name: csi-proxy-disk-v1
                  mountPath: \\.\pipe\csi-proxy-disk-v1
                - name: csi-proxy-system-v1alpha1
                  mountPath: \\.\pipe\csi-proxy-system-v1alpha1
              ports:
                - name: healthz
                  containerPort: 9808
                  protocol: TCP
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: healthz
                initialDelaySeconds: 10
                timeoutSeconds: 5
                periodSeconds: 5
                failureThreshold: 3
            - name: liveness-probe
              image: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0
              args:
                - "--v=4"
                - "--csi-address=/csi/csi.sock"
              volumeMounts:
                - name: plugin-dir
                  mountPath: /csi
          volumes:
            - name: registration-dir
              hostPath:
                path: 'C:\var\lib\kubelet\plugins_registry\'
                type: Directory
            - name: plugin-dir
              hostPath:
                path: 'C:\var\lib\kubelet\plugins\csi.vsphere.vmware.com\'
                type: DirectoryOrCreate
            - name: pods-mount-dir
              hostPath:
                path: \var\lib\kubelet
                type: Directory
            - name: csi-proxy-disk-v1
              hostPath:
                path: \\.\pipe\csi-proxy-disk-v1
                type: ''
            - name: csi-proxy-volume-v1
              hostPath:
                path: \\.\pipe\csi-proxy-volume-v1
                type: ''
            - name: csi-proxy-filesystem-v1
              hostPath:
                path: \\.\pipe\csi-proxy-filesystem-v1
                type: ''
            - name: csi-proxy-system-v1alpha1
              hostPath:
                path: \\.\pipe\csi-proxy-system-v1alpha1
                type: ''
          tolerations:
            - effect: NoExecute
              operator: Exists
            - effect: NoSchedule
              operator: Exists
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-vsphere-csi-node-windows
  namespace: ${NAMESPACE}
---
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}-cloud-controller-manager
  namespace: ${NAMESPACE}
stringData:
  data: |
    apiVersion: v1
    kind: ServiceAccount
    metadata: 
      labels: 
        component: cloud-controller-manager
        vsphere-cpi-infra: service-account
      name: cloud-controller-manager
      namespace: kube-system
type: addons.cluster.x-k8s.io/resource-set
---
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}-cloud-provider-vsphere-credentials
  namespace: ${NAMESPACE}
stringData:
  data: |
    apiVersion: v1
    kind: Secret
    metadata: 
      labels: 
        component: cloud-controller-manager
        vsphere-cpi-infra: secret
      name: cloud-provider-vsphere-credentials
      namespace: kube-system
    stringData: 
      ${VSPHERE_SERVER}.password: ${VSPHERE_PASSWORD}
      ${VSPHERE_SERVER}.username: ${VSPHERE_USERNAME}
    type: Opaque
type: addons.cluster.x-k8s.io/resource-set
---
apiVersion: v1
data:
  data: |
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata: 
      labels: 
        component: cloud-controller-manager
        vsphere-cpi-infra: role
      name: system:cloud-controller-manager
    rules: 
    - apiGroups: 
      - ""
      resources: 
      - events
      verbs: 
      - create
      - patch
      - update
    - apiGroups: 
      - ""
      resources: 
      - nodes
      verbs: 
      - '*'
    - apiGroups: 
      - ""
      resources: 
      - nodes/status
      verbs: 
      - patch
    - apiGroups: 
      - ""
      resources: 
      - services
      verbs: 
      - list
      - patch
      - update
      - watch
    - apiGroups: 
      - ""
      resources: 
      - services/status
      verbs: 
      - patch
    - apiGroups: 
      - ""
      resources: 
      - serviceaccounts
      verbs: 
      - create
      - get
      - list
      - watch
      - update
    - apiGroups: 
      - ""
      resources: 
      - persistentvolumes
      verbs: 
      - get
      - list
      - watch
      - update
    - apiGroups: 
      - ""
      resources: 
      - endpoints
      verbs: 
      - create
      - get
      - list
      - watch
      - update
    - apiGroups: 
      - ""
      resources: 
      - secrets
      verbs: 
      - get
      - list
      - watch
    - apiGroups: 
      - coordination.k8s.io
      resources: 
      - leases
      verbs: 
      - get
      - watch
      - list
      - update
      - create
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata: 
      labels: 
        component: cloud-controller-manager
        vsphere-cpi-infra: cluster-role-binding
      name: system:cloud-controller-manager
    roleRef: 
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:cloud-controller-manager
    subjects: 
    - kind: ServiceAccount
      name: cloud-controller-manager
      namespace: kube-system
    - kind: User
      name: cloud-controller-manager
    ---
    apiVersion: v1
    data: 
      vsphere.conf: |
        global: 
          port: 443
          secretName: cloud-provider-vsphere-credentials
          secretNamespace: kube-system
          thumbprint: '${VSPHERE_TLS_THUMBPRINT}'
        vcenter: 
          ${VSPHERE_SERVER}:
            datacenters: 
            - '${VSPHERE_DATACENTER}'
            server: '${VSPHERE_SERVER}'
    kind: ConfigMap
    metadata: 
      name: vsphere-cloud-config
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata: 
      labels: 
        component: cloud-controller-manager
        vsphere-cpi-infra: role-binding
      name: servicecatalog.k8s.io:apiserver-authentication-reader
      namespace: kube-system
    roleRef: 
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: extension-apiserver-authentication-reader
    subjects: 
    - kind: ServiceAccount
      name: cloud-controller-manager
      namespace: kube-system
    - kind: User
      name: cloud-controller-manager
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata: 
      labels: 
        component: cloud-controller-manager
        tier: control-plane
      name: vsphere-cloud-controller-manager
      namespace: kube-system
    spec: 
      selector: 
        matchLabels: 
          name: vsphere-cloud-controller-manager
      template: 
        metadata: 
          labels: 
            component: cloud-controller-manager
            name: vsphere-cloud-controller-manager
            tier: control-plane
        spec: 
          affinity: 
            nodeAffinity: 
              requiredDuringSchedulingIgnoredDuringExecution: 
                nodeSelectorTerms: 
                - matchExpressions: 
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
                - matchExpressions: 
                  - key: node-role.kubernetes.io/master
                    operator: Exists
          containers: 
          - args: 
            - --v=2
            - --cloud-provider=vsphere
            - --cloud-config=/etc/cloud/vsphere.conf
            image: gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.25.3
            name: vsphere-cloud-controller-manager
            resources: 
              requests: 
                cpu: 200m
            volumeMounts: 
            - mountPath: /etc/cloud
              name: vsphere-config-volume
              readOnly: true
          hostNetwork: true
          priorityClassName: system-node-critical
          securityContext: 
            runAsUser: 1001
          serviceAccountName: cloud-controller-manager
          tolerations: 
          - effect: NoSchedule
            key: node.cloudprovider.kubernetes.io/uninitialized
            value: "true"
          - effect: NoSchedule
            key: node-role.kubernetes.io/master
            operator: Exists
          - effect: NoSchedule
            key: node-role.kubernetes.io/control-plane
            operator: Exists
          - effect: NoSchedule
            key: node.kubernetes.io/not-ready
            operator: Exists
          volumes: 
          - configMap: 
              name: vsphere-cloud-config
            name: vsphere-config-volume
      updateStrategy: 
        type: RollingUpdate
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-cpi-manifests
  namespace: ${NAMESPACE}




---
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSet
metadata:
  name: ${CLUSTER_NAME}-calico-module-resource
  namespace: ${NAMESPACE}
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  resources:
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-calico-module-cr
  strategy: Reconcile
---
apiVersion: v1
data:
  calico.yaml: |
    apiVersion: platform.verrazzano.io/v1alpha1
    kind: Module
    metadata:
      name: calico
      namespace: default
    spec:
      moduleName: calico
      targetNamespace: default
      values:
        tigeraOperator:
          version: ${TIGERA_TAG=v1.29.0}
        installation:
          cni:
            type: Calico
          calicoNetwork:
            bgp: Disabled
            ipPools:
              - cidr: ${POD_CIDR=192.168.0.0/16}
                encapsulation: VXLAN
          registry: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}
          imagePath: ${OCNE_IMAGE_PATH=olcne}
kind: ConfigMap
metadata:
  annotations:
    note: generated
  labels:
    type: generated
  name: ${CLUSTER_NAME}-calico-module-cr
  namespace: ${NAMESPACE}

Generate and apply the template by running the following command:
```
$ clusterctl generate yaml --from vsphere-capi.yaml | kubectl apply -f -
```

To get the kubeconfig file, run:

$ clusterctl get kubeconfig kluster1 -n kluster1 > kluster1

Finish cluster configuration

After the cluster resources are created, you must perform some additional steps to finish the configuration of the cluster.

If vSphere does not have a load balancer, then you can deploy MetalLB.
```
$ export KUBECONFIG=kluster1
 
ADDRESS_RANGE=${1:-"subnet-from-vSphere-network"};
 
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml --wait=true;
$ kubectl rollout status deployment -n metallb-system controller -w;
$ kubectl apply -f -  <<EOF1
  apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  metadata:
    name: vzlocalpool
    namespace: metallb-system
  spec:
    addresses:
    - ${ADDRESS_RANGE}
EOF1
 
$ kubectl apply -f -  <<-EOF2
  apiVersion: metallb.io/v1beta1
  kind: L2Advertisement
  metadata:
    name: vzmetallb
    namespace: metallb-system
  spec:
    ipAddressPools:
    - vzlocalpool
EOF2
 
$ sleep 10;
$ kubectl wait --namespace metallb-system --for=condition=ready pod --all --timeout=300s
```
Create a default storage class on the cluster.
```
$ export KUBECONFIG=kluster1
$ kubectl apply -f -  <<-EOF
  kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
    name: vmware-sc
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
  provisioner: csi.vsphere.vmware.com
  volumeBindingMode: WaitForFirstConsumer
EOF
```
Install Verrazzano on the managed cluster.
```
$ export KUBECONFIG=kluster1 

$ vz install -f - <<EOF
  apiVersion: install.verrazzano.io/v1beta1
  kind: Verrazzano
  metadata:
    name: example-verrazzano
  spec:
    profile: dev
    defaultVolumeSource:
      persistentVolumeClaim:
        claimName: verrazzano-storage
    volumeClaimSpecTemplates:
      - metadata:
          name: verrazzano-storage
        spec:
          resources:
            requests:
              storage: 2Gi
EOF
```

Your admin cluster and first managed cluster are now up and running and ready to deploy applications. You can also add more managed clusters.

For more information, refer to the documentation for Cluster API and Cluster API vSphere:

Troubleshoot the deployment

If the deployment of the vSphere resources fails, then you can check the log files to diagnose the issue.

The vSphere cluster controller provider logs:

$ kubectl logs -n verrazzano-capi -l cluster.x-k8s.io/provider=infrastructure-vsphere

The OCNE control plane provider logs:

$ kubectl logs -n verrazzano-capi -l cluster.x-k8s.io/provider=control-plane-ocne

NOTE: If the CSI pod deploys before Calico, then the pod may enter a CrashLoop state. Restart the pod to fix the issue.

$ kubectl --kubeconfig kluster1 scale deploy  -n kube-system vsphere-csi-controller --replicas=0
$ kubectl --kubeconfig kluster1 scale deploy  -n kube-system vsphere-csi-controller --replicas=1

Delete the clusters

Delete the managed clusters.
```
$ kubectl delete cluster $CLUSTER_NAME
```
Delete the admin cluster.
```
$ kind delete cluster
```

Do not use kubectl delete -f capi-quickstart.yaml to delete the entire cluster template at once because it might leave behind pending resources that you need to clean up manually.