Using Oracle Cloud Infrastructure Storage

3 Using Oracle Cloud Infrastructure Storage

The Oracle Cloud Infrastructure Cloud Controller Manager module implements a Container Storage Interface (CSI) plugin for Kubernetes clusters that provides you with the following storage services for creating persistent data storage:

Oracle Cloud Infrastructure Block Volume service

The Oracle Cloud Infrastructure Block Volume service provides dynamically provisioned and managed block storage volumes. The block storage volumes can be created in sizes ranging from 50 GB to 32 TB.

After you attach a volume to a Kubernetes application, the volume can be used in a similar way to a regular hard drive. The volume can also be disconnected and attached to another instance without the loss of data.

Block Volume volumes are automatically replicated to help protect against data loss.

To read about example use cases, and for more information about the Oracle Cloud Infrastructure Block Volume service, see the Oracle Cloud Infrastructure documentation.

Oracle Cloud Infrastructure File Storage service

The Oracle Cloud Infrastructure File Storage service provides a scalable and durable enterprise-grade network file system. The File Storage service uses the Network File System version 3.0 (NFSv3) protocol. The service uses the Network Lock Manager (NLM) protocol for file locking functionality.

The File Storage service transparently manages storage provisioning as data storage needs increase from the order of bytes to exabytes.

Large compute clusters of thousands of instances can use the File Storage service for high-performance shared storage.

Oracle Cloud Infrastructure File Storage uses 5-way replicated storage, in different fault domains, to provide redundancy for resilient data protection.

To read about example use cases, and for more information about the Oracle Cloud Infrastructure File Storage service, see the Oracle Cloud Infrastructure documentation

The following sections provide examples to illustrate how to use each of the storage services to set up persistent storage for Kubernetes applications in Oracle Cloud Native Environment on Oracle Cloud Infrastructure instances.

Creating Block Storage

This section contains a basic test to verify you can create Oracle Cloud Infrastructure block storage to provide persistent storage to applications running on Kubernetes.

To create a test application to use Oracle Cloud Infrastructure block storage:

Create a Kubernetes PersistentVolumeClaim file. On a control plane node, create a file named pvc.yaml. Copy the following into the file.
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: myoci-pvc
spec:
 accessModes:
  - ReadWriteOnce
 storageClassName: oci-bv
 resources:
  requests:
    storage: 50Gi
```
Note that the accessModes setting for Oracle Cloud Infrastructure storage must be ReadWriteOnce. The minimum Oracle Cloud Infrastructure block size is 50Gi.
Create the Kubernetes PersistentVolumeClaim.
```
kubectl apply -f pvc.yaml
```

You can see the PersistentVolumeClaim is created using the kubectl get pvc command:

kubectl get pvc

The output looks similar to:

NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myoci-pvc   Pending                                      oci-bv         15s

The STATUS is Pending and means the claim is waiting for an application to claim it.

You can get more details about the PersistentVolumeClaim using the kubectl describe pvc command. For example:

kubectl describe pvc myoci-pvc

The output looks similar to:

Name:          myoci-pvc
Namespace:     default
StorageClass:  oci-bv
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                     From                         ...
  ----    ------                ----                    ----                         
  Normal  WaitForFirstConsumer  2m18s (x26 over 8m29s)  persistentvolume-controller  ...

Create a Kubernetes application that uses the PersistentVolumeClaim. Create a file named nginx.yaml and copy the following into the file.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: mynginx
  name: mynginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: mynginx
  template:
    metadata:
      labels:
        run: mynginx
    spec:
      containers:
      - image: container-registry.oracle.com/olcne/nginx:1.17.7 
        name: mynginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-pvc
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-pvc
        persistentVolumeClaim:
          claimName: myoci-pvc

Start the application:
```
kubectl apply -f nginx.yaml
```
You can see the application is running using the kubectl get deployment command:
```
kubectl get deployment
```
The output looks similar to:
```
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
mynginx   1/1     1            1           63s
```
You can see the application is using the PersistentVolumeClaim to provide persistent storage on Oracle Cloud Infrastructure using the kubectl describe deployment command:
```
kubectl describe deployment mynginx
```
The output looks similar to:
```
...
Pod Template:
  Labels:  run=mynginx
  Containers:
   mynginx:
    Image:        container-registry.oracle.com/olcne/nginx:1.17.7
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /usr/share/nginx/html from nginx-pvc (rw)
  Volumes:
   nginx-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  myoci-pvc
    ReadOnly:   false
...
```
Note the ClaimName is myoci-pvc, which is the name of the PersistentVolumeClaim created earlier.

You can see the PersistentVolumeClaim is now bound to this application using the kubectl get pvc command:
```
kubectl get pvc
```
The output looks similar to:
```
NAME        STATUS   VOLUME             CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myoci-pvc   Bound    csi-84175067-...   50Gi       RWO            oci-bv         1m
```
Tip:

If you sign in to Oracle Cloud Infrastructure, you can see a block volume created with the name listed in the VOLUME column. The block volume is attached to the compute instance on which the Kubernetes application is running.
You can delete the test application using:
```
kubectl delete deployment mynginx
```
You can delete the PersistentVolumeClaim using:
```
kubectl delete pvc myoci-pvc
```
The storage is deleted.

Tip:

If you sign in to Oracle Cloud Infrastructure, you can see the block volume is terminated.

Creating File Storage

This section contains a basic example to verify you can use an Oracle Cloud Infrastructure File Storage file system to provide persistent file storage to applications running on Kubernetes. The example involves:

Creating a File Storage file system in Oracle Cloud Infrastructure.
Using YAML files to create a PersistentVolume and a PersistentVolumeClaim for the File Storage file system.
Using a YAML file to create a pod with a mount to the volume to write a test file in the File Storage file system.

Setting up a File System

Create a File Storage file system in a Virtual Cloud Network (VCN) in Oracle Cloud Infrastructure.

Tip:

For a step-by-step guide on provisioning a File System in Oracle Cloud Infrastructure, see the Oracle Luna Lab Provision Persistent Volumes Using File Storage Service on Oracle Cloud Native Environment.

To create a file system:

Configure a VCN for the File Storage file system. The precise configuration of the VCN varies. For example scenarios, see the Oracle Cloud Infrastructure documentation.
Create a File Storage file system. For more information, see the Oracle Cloud Infrastructure documentation.
Create a File Storage mount target to enable network access to the file system. For more information, see the Oracle Cloud Infrastructure documentation.
In Oracle Cloud Infrastructure, find and make a note of the following file system attributes. These are needed when you create YAML files or mount the file system in this example:
- The OCID of the file system.
- The export path of the file system. This example assumes you're setting the export path to /my-fss-export.
- The mount commands. These commands are provided on the Mount Commands page of the File System. You use these commands later in the example when mounting the exported file system.
- The IP address of the mount target.

Using a File System

To create a test application to use an Oracle Cloud Infrastructure File Storage file system:

Create a PersistentVolume file. On a control plane node, create a file called fss-pv.yaml with the following contents:
```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: fss-pv
spec:
  capacity:
    storage: 50Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: fss.csi.oraclecloud.com
    volumeHandle: <filesystem_OCID>:<mount_target_IP>:<mount_target_export_path>
```
The accessModes for the file system must be ReadWriteMany.

The storage option is a Kubernetes requirement and must be included. The File Storage service ignores this value and creates a new file system with a default size, regardless of the value you specify for storage.

The volumeHandle consists of a colon separated list of File Storage file system attributes, noted in an earlier step. For example:

ocid1.filesystem.oc1.iad.aaaa...:10.0.0.200:/my-fss-export
Create the PersistentVolume.
```
kubectl apply -f fss-pv.yaml
```

Get a list of the PersistentVolumes.

kubectl get pv

The output looks similar to:

NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
fss-pv   50Gi       RWX            Retain           Available                                   47s

Create a PersistentVolumeClaim file. Create a file called fss-pvc.yaml. Copy the following into the file:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fss-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: fss-pv
```
The accessModes for the file system must be ReadWriteMany.

The storage option is a Kubernetes requirement and must be included. The File Storage service ignores this value and creates a new file system with a default size, regardless of the value you specify for storage.

The volumeName is set to the name attribute in the PersistentVolume file fss-pv.yaml.
Create the PersistentVolumeClaim.
```
kubectl apply -f fss-pvc.yaml
```

Get a list of PersistentVolumeClaims.

kubectl get pvc

The output looks similar to:

NAME      STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
fss-pvc   Bound    fss-pv   50Gi       RWX                           33s

Create a pod manifest file. Create a file called fss-pod.yaml. Copy the following into the file:
```
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: container-registry.oracle.com/os/oraclelinux:9-slim
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: fss-pvc
```
The image option specifies the registry location to an Oracle Linux container image.

The mountPath option specifies the directory mounted to the persistent storage.

The command option specifies a command to write to file out.txt in the /data directory.
Create the pod.
```
kubectl apply -f fss-pod.yaml
```

Get a list of the pods.

kubectl get pods

The output looks similar to:

NAME   READY   STATUS    RESTARTS   AGE
app    1/1     Running   0          21s

Open a shell to the container within the pod:

kubectl exec -i -t app --container app -- /bin/bash

From the container's shell, confirm the container is writing to the /data/out.txt file by using the tail command:
```
tail -f /data/out.txt
```
The date and time output of the date -u command is listed to the terminal and written to the /data/out.txt file on the container. Exit the tail program by using CTRL+C.
Exit the container shell using the exit command:
```
exit
```
Verify that the /data directory within the container is mounted to the file system export path. This step uses the mount commands noted when you created the file system.
1. On the control plane node, install the NFS client.
```
sudo dnf install nfs-utils
```
2. Create and mount a local directory to the file system's export path.
```
sudo mkdir -p /mnt/my-fss-export
sudo mount 10.0.0.200:/my-fss-export /mnt/my-fss-export
```
  Replace 10.0.0.200 with the IP address of the mount target.
3. Confirm the output.txt file the container is writing to is in the file system's export path. You can use the tail command for this:
```
sudo tail -f /mnt/my-fss-export/out.txt
```
  The date and time being written to the file by the pod is listed. Exit the tail program by using CTRL+C.
Delete the pod. If you don't see the command prompt after the pod is deleted you can use the CTRL+C key combination to get it back:
```
kubectl delete pod app
```
Delete the PersistentVolumeClaim:
```
kubectl delete pvc fss-pvc
```

Confirm that the status of the PersistentVolume is Released:

kubectl get pv

The output looks similar to:

NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM             STORAGECLASS   REASON   AGE
fss-pv   50Gi       RWX            Retain           Released   default/fss-pvc                           57m

Delete the PersistentVolume:
```
kubectl delete pv fss-pv
```
Confirm the file is still present:
```
ls -l  /mnt/my-fss-export/out.txt
```
Remove the mount:
```
sudo umount /mnt/my-fss-export
```
Confirm the out.txt file is no longer available and the mount is removed:
```
ls -l /mnt/my-fss-export
```
Confirm the /mnt/my-fss-export is no longer mounted:
```
mount | grep my-fss-export
```