Oracle Linux Cloud Native Environment (OLCNE)

Storage

Verrazzano requires persistent storage for several components. This persistent storage is provided by a default storage class. A number of persistent storage providers exist for Kubernetes. This guide will focus on pre-allocated persistent volumes. In particular, the provided samples will illustrate the use of OCI’s NFS File System.

OCI example

Before storage can be exposed to Kubernetes, it must be created. In OCI, this is done using File System resources. Using the OCI Console, create a new File System. Within the new File System, create an Export. Remember the value used for Export Path as it will be used later. Also note the Mount Target’s IP Address for use later.

After the exports have been created, referenced persistent volume folders (for example, /example/pv0001) will need to be created. In OCI, this can be done by mounting the export on one of the Kubernetes worker nodes and creating the folders. In the following example, the value /example is the Export Path and 10.0.1.8 is the Mount Target’s IP Address. The following command should be run on one of the Kubernetes worker nodes. This will result in the creation of nine persistent volume folders. The reason for nine persistent volume folders is covered in the next section.

$ sudo mount 10.0.1.8:/example /mnt
$ for x in {0001..0009}; do sudo mkdir -p /mnt/pv${x} && sudo chmod 777 /mnt/pv${x}; done

Persistent Volumes

A default Kubernetes storage class is required by Verrazzano. When using pre-allocated PersistentVolumes, for example NFS, persistent volumes should be declared as following. The value for name may be customized but will need to match the PersistentVolume storageClassName value later.

• Create a default StorageClass

  $ cat << EOF | kubectl apply -f -
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: example-nfs
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
    provisioner: kubernetes.io/no-provisioner
    volumeBindingMode: WaitForFirstConsumer
  EOF
  

• Create the required number of PersistentVolume resources. The Verrazzano system requires five persistent volumes for itself. The following command creates nine persistent volumes. The value for storageClassName must match the above StorageClass name. The values for name may be customized. The value for path must match the Export Path of the Export from above, combined with the persistent volume folder from above. The value for server must be changed to match the location of your file system server.

  $ for n in {0001..0009}; do cat << EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv${n}
    spec:
      storageClassName: example-nfs
      accessModes:
        - ReadWriteOnce
        - ReadWriteMany
      capacity:
        storage: 50Gi
      nfs:
        path: /example/pv${n}
        server: 10.0.1.8
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Recycle
  EOF
  done
  

Load Balancers

Verrazzano on Oracle Linux Cloud Native Environment uses external load balancer services. These will not automatically be provided by Verrazzano or Kubernetes. Two load balancers must be deployed outside of the subnet used for the Kubernetes cluster. One load balancer is for management traffic and the other for application traffic.

Specific steps will differ for each load balancer provider, but a generic configuration and an OCI example follow.

Generic configuration:

• Target Host: Host names of Kubernetes worker nodes
• Target Ports: See table
• External Ports: See table
• Distribution: Round-robin
• Health Check: TCP

OCI example

The following details can be used to create OCI load balancers for accessing application and management user interfaces, respectively. These load balancers will route HTTP/HTTPS traffic from the Internet to the private subnet. If load balancers are desired, then they should be created now even though the application and management endpoints will be installed later.

• Application Load Balancer: Public Subnet
  • Listeners
    • HTTP Listener: Protocol TCP, Port 80
    • HTTPS Listener: Protocol TCP, Port 443
  • Backend Sets
    • HTTP Backend Sets:
      • Health Check: Protocol TCP, Port 31380
      • Backends: Kubernetes Worker Nodes, Port 31380, Distribution Policy Weighted Round Robin
    • HTTPS Backend Sets
      • Health Check: Protocol TCP, Port 31390
      • Backends: Kubernetes Worker Nodes, Port 31390, Distribution Policy Weighted Round Robin
• Management Load Balancer: Public Subnet
  • Listeners
    • HTTP Listener: Protocol TCP, Port 80
    • HTTPS Listener: Protocol TCP, Port 443
  • Backend Sets
    • HTTP Backend Sets:
      • Health Check: Protocol TCP, Port 30080
      • Backends: Kubernetes Worker Nodes, Port 30080, Distribution Policy Weighted Round Robin
    • HTTPS Backend Sets
      • Health Check: Protocol TCP, Port 30443
      • Backends: Kubernetes Worker Nodes, Port 30443, Distribution Policy Weighted Round Robin

DNS

When using the Verrazzanospec.components.dns.external DNS type, the installer searches the DNS zone you provide for two specific A records. These are used to configure the cluster and should refer to external addresses of the load balancers in the previous step. The A records will need to be created manually.

NOTE: At this time, the only supported deployment for Oracle Linux Cloud Native Environment is the external DNS type.

For example:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

Verrazzano installation will result in a number of management services that need to point to the ingress-mgmt address.

keycloak.myenv.example.com                      CNAME   ingress-mgmt.myenv.example.com.
rancher.myenv.example.com                       CNAME   ingress-mgmt.myenv.example.com.

grafana.vmi.system.myenv.example.com            CNAME   ingress-mgmt.myenv.example.com.
prometheus.vmi.system.myenv.example.com         CNAME   ingress-mgmt.myenv.example.com.
kibana.vmi.system.myenv.example.com             CNAME   ingress-mgmt.myenv.example.com.
elasticsearch.vmi.system.myenv.example.com      CNAME   ingress-mgmt.myenv.example.com.

For simplicity, an administrator may want to create wildcard DNS records for the management addresses:

*.system.myenv.example.com                      CNAME   ingress-mgmt.myenv.example.com.

OR

*.myenv.example.com                             CNAME   ingress-mgmt.myenv.example.com.

OCI example

DNS is configured in OCI by creating DNS zones in the OCI Console. When creating a DNS zone, use these values:

• Method: Manual
• Zone Name: <dns-suffix>
• Zone Type: Primary

The value for <dns-suffix> excludes the environment (for example, use the example.com portion of myenv.example.com).

DNS A records must be manually added to the zone and published using values described above. DNS CNAME records, in the same way.