Note:

Deploy Longhorn for Kubernetes on OKE

Introduction

Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) is a fully-managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. You specify the compute resources that your applications require, and OKE provisions them on Oracle Cloud Infrastructure (OCI) in an existing OCI tenancy. Container Engine for Kubernetes uses Kubernetes - the open-source system for automating deployment, scaling, and management of containerized applications across clusters of hosts.

The OCI Block Volume service provides persistent, durable, and high-performance block storage for your data and OKE can utilize the block volumes as persistent disks for your Kubernetes environment and is fully managed by OKE. In case you want to be in complete control of the persistent storage solution you can deploy Longhorn on OKE and use it as your storage class. It gives you complete control over persistent volumes, scaling, backups and scheduling.

Objective

Prerequisites

Task 1: Create an OKE cluster

  1. Log in to the OCI Console, navigate to Oracle Container Engine for Kubernetes and click Create.

  2. In the Create cluster wizard, click Custom Create.

    Custom Create

  3. Specify the Cluster name, select the Kubernetes version you want to use and click Next at the bottom of the wizard.

    Create Cluster

    Note: You must set up the VCN, subnets and routes before starting with the cluster creation

    • You can choose either of the networking types. The VCN native networking provides better performance. If you choose VCN native networking make sure your subnet has enough IP addresses available.

      Networking Type

    • Configure the node pool, choose your desired compute resources. In Advanced options, paste the below custom cloud-init script.

      cloud-init script

    • OKE does not provide an option to modify the node template, so in order to attach a block volume, you must run a cloud-init script at node initialization. The script will create a block volume of specified size and performance and attach it to the node at initialization.

    • Make sure to modify the size_in_gbs (size of block storage to attach at init), vpus_per_gb (vpu per gb for the block storage attachment), and mode(attachment mode, PARA or ISCSI) variables in the script as per your need.

    You need to create a dynamic group and provide this group manage access to block storage, this allows us to use instance principal authentication in our script.

    cloud-init script:

    #!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-init.sh
bash /var/run/oke-init.sh

echo "installing python3-pip , oci sdk\n"
sudo yum install python3 -y
sudo yum install python3-pip -y
pip3 install oci
pip3 install requests

cat << EOF > pyscript.py
#!/usr/bin/python

import oci
import requests

size_in_gbs = 200
vpus_per_gb = 10
mode = 'PARA'
device_path = "/dev/oracleoci/oraclevdb"
signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()
compute_client = oci.core.ComputeClient({}, signer = signer)
block_storage_client = oci.core.BlockstorageClient({}, signer = signer)

def get_current_instance_details():
    r = requests.get(url= 'http://169.254.169.254/opc/v1/instance')
    return r.json()

def create_volume(block_storage, compartment_id, availability_domain, display_name: str):
    print("--- creating block volume ---")
    result = block_storage.create_volume(
        oci.core.models.CreateVolumeDetails(
            compartment_id=compartment_id,
            availability_domain=availability_domain,
            display_name=display_name,
            size_in_gbs = size_in_gbs,
            vpus_per_gb = vpus_per_gb
        )
    )
    volume = oci.wait_until(
        block_storage,
        block_storage.get_volume(result.data.id),
        'lifecycle_state',
        'AVAILABLE'
    ).data
    print('--- Created Volume ocid: {} ---'.format(result.data.id))

    return volume

def attach_volume(instance_id, volume_id,device_path):
    volume_attachment_response = ""
    if mode == 'ISCSI':
        print("--- Attaching block volume {} to instance {}---".format(volume_id,instance_id))
        volume_attachment_response = compute_client.attach_volume(
            oci.core.models.AttachIScsiVolumeDetails(
                display_name='IscsiVolAttachment',
                instance_id=instance_id,
                volume_id=volume_id,
                device= device_path
                )
            )
    elif mode == 'PARA':
        volume_attachment_response = compute_client.attach_volume(
            oci.core.models.AttachParavirtualizedVolumeDetails(
            display_name='ParavirtualizedVolAttachment',
            instance_id=instance_id,
            volume_id=volume_id,
            device= device_path
        )
    )

    oci.wait_until(
        compute_client,
        compute_client.get_volume_attachment(volume_attachment_response.data.id),
        'lifecycle_state',
        'ATTACHED'
    )
    print("--- Attaching complete block volume {} to instance {}---".format(volume_id,instance_id))
    print(volume_attachment_response.data)

    # Call instance metadata uri to get current instace details
    instanceDetails = get_current_instance_details()
    print(instanceDetails)
    volume = create_volume(block_storage= block_storage_client, compartment_id= instanceDetails['compartmentId'], availability_domain=instanceDetails['availabilityDomain'], display_name= instanceDetails['displayName'])
    attach_volume(instance_id=instanceDetails['id'], volume_id=volume.id, device_path= device_path)

    EOF

    echo "running python script\n"
    chmod 755 pyscript.py
    ./pyscript.py

    echo "creating file system on volume\n"
    sudo /sbin/mkfs.ext4 /dev/oracleoci/oraclevdb
    echo "mounting volume\n"
    sudo mkdir /mnt/volume
    sudo mount /dev/oracleoci/oraclevdb /mnt/volume
    echo "adding entry to fstab\n"
    echo "/dev/oracleoci/oraclevdb /mnt/volume ext4 defaults,_netdev,nofail 0 2" |  sudo tee -a /etc/fstab

  4. Review and click Create Cluster and wait for the cluster to become available.

  5. Once the cluster is available, go to Node pools and then Nodes. You can see the nodes are in ready state, click on any node, it will open the instance details page, go to attached Block Volumes and you can verify that a block volume (of size and vpu as mentioned in the cloud-init script) is attached to the instance.

    Block Volume

Task 2: Set up Longhorn

  1. Once the cluster is available, you can access the cluster using cloud shell, click Access Cluster and copy and run the command in cloud shell.

    Access Cluster

  2. Run kubectl get node to get the list of schedulable nodes.

    Get nodes

  3. In order for Longhorn to recognize the attached disk we need to add labels and annotations to the Nodes as mentioned in Longhorn default disk setup. To do this we are going to create a patch.yaml file to patch the nodes.

    metadata:
labels:
    node.longhorn.io/create-default-disk: "config"
annotations:
    node.longhorn.io/default-disks-config: '[
    {
        "path":"/var/lib/longhorn",
        "allowScheduling":true
    },
    {
        "path":"/mnt/volume",
        "allowScheduling":true
    }
]'

  4. Run the following command for each node.

    kubectl patch node <node name> --patch-file patch.yaml

    Patch Node

  5. Run describe node command to verify that the nodes are patched.

    kubectl describe node <node name>

    Describe Node

Task 3: Install Longhorn

  1. In this tutorial, we use Helm to deploy Longhorn on the OKE cluster. Follow the instructions mentioned in this document: Install with Helm.

  2. Run the following command to deploy Longhorn.

    helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.3.2 --set defaultSettings.createDefaultDiskLabeledNodes=true

    Deploy Longhorn

    Note: Set the defaultSettings.createDefaultDiskLabeledNodes property to true while deploying. This will tell longhorn to use the attach block volume config we provided earlier.

  3. Verify that the pods are created and running.

    Check Pods

  4. To enable the UI for a better view of Longhorn and its administration, install the Ingress Controller as mentioned in this document: Example: Setting Up an Ingress Controller on a Cluster.

  5. Create an ingress.yaml file to expose the Longhorn UI.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: longhorn-ingress
namespace: longhorn-system
annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
    paths:
        - path: /
        pathType: Prefix
        backend:
            service:
            name: longhorn-frontend
            port:
                number: 80

  6. Run kubectl apply -f ingress.yaml to expose the Longhorn UI using Ingress gateway.

    Ingress Gateway

  7. Get the gateway url by running the following command.

    kubectl get ingress -n longhorn-system.

    Ingress IP

  8. Open the IP address from a browser to access the Longhorn Console and confirm it is live.

    Longhorn UI

    You can see that we have the storage available and ready to use.

Task 4: Scale Longhorn

Longhorn gives you complete control of the storage solution for your kubernetes deployments, but scaling longhorn is still a manual process, there are 3 ways you can scale the Longhorn setup.

  1. Increase the size or the performance units of the attached block volumes: This is a manual process, you have to scale each block volume individually and run some scripts on the nodes to extend the storage. For more information, see Resizing a Volume.

  2. Attach more volumes: You have to create and attach volumes to the nodes manually.

  3. Scale the cluster by increasing the nodes: Since we have already provided a cloud-init script, it will create the block volume and attach it to the node. We must patch the node after it is ready as mentioned in Task 2.3 for Longhorn to recognize the attached storage.

Note: You can modify the script to attach the desired number of block volumes to a single node.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.