Provisioning PVCs on the File Storage with Lustre Service

To dynamically provision a PVC on a new Lustre file system dynamically created by the CSI volume plugin in the File Storage with Lustre service:

1. Create security rules in either a network security group (recommended) or a security list for both the Lustre file system, and for the cluster's worker nodes subnet.

   The security rules to create depend on the relative network locations of the Lustre file system and the worker nodes which act as the client, according to the following scenarios:

   • Option 1: Client and Lustre in Different Subnets
   • Option 2: Client and Lustre in the Same Subnet

   These scenarios, the security rules to create, and where to create them, are fully described in the File Storage with Lustre service documentation (see Required VCN Security Rules).

2. Define a new storage class that uses the lustre.csi.oraclecloud.com provisioner:
   1. Create a manifest file (for example, in a file named lustre-dyn-st-class.yaml), specify a name for the new storage class, and specify values for required and optional parameters:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: <storage-class-name>
      provisioner: lustre.csi.oraclecloud.com
      parameters:
        availabilityDomain: <ad-name>
        compartmentId: <compartment-ocid>   # optional
        subnetId: <subnet-ocid>
        performanceTier: <value>
        fileSystemName: <name>               # optional
        kmsKeyId: <key-ocid>                 # optional
        nsgIds: '["<nsg-ocid>"]'             # optional
        rootSquashEnabled: "<true | false>"  # optional
        rootSquashUid: "<value>"             # optional
        rootSquashGid: "<value>"             # optional
        rootSquashClientExceptions: '["<ip-address>"]'   # optional
        oci.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}'
        oci.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}'
        setupLnet: "<true | false>"                    # optional
        lustreSubnetCidr: "<cidr-block>"      # optional
        lustrePostMountParameters: '[{"<parameter1>": <value>},{"<parameter2>": <value>}]' # optional

      where:

      • name: <storage-class-name>: Required. A name of your choice for the storage class.
      • availabilityDomain: <ad-name>: Required. The name of the availability domain in which to create the new Lustre file system. For example, availabilityDomain: US-ASHBURN-AD-1. To find out the availability domain name to use, run the oci iam availability-domain list CLI command (or use the ListAvailabilityDomains operation). For more information, see Your Tenancy's Availability Domain Names.
      • compartmentId: <compartment-ocid>: Optional. The OCID of the compartment to which the new Lustre file system is to belong. If not specified, defaults to the same compartment as the cluster. For example, compartmentId: ocid1.compartment.oc1..aaa______t6q
      • subnetId: <subnet-ocid>: Required. The OCID of the subnet in which to mount the new Lustre file system. For example, subnetId: ocid1.subnet.oc1.iad.aaaa______kfa
      • performanceTier: <value>: Required. The Lustre file system performance tier. Allowed values:
        • MBPS_PER_TB_125
        • MBPS_PER_TB_250
        • MBPS_PER_TB_500
        • MBPS_PER_TB_1000
      • fileSystemName: <name>: Optional. The Lustre file system name, up to 8 characters. If not specified, a default value will be randomly generated and used. For example, fileSystemName: aiworkfs
      • kmsKeyId: <key-ocid>: Optional. The OCID of a master encryption key that you manage, with which to encrypt data at rest. If not specified, data is encrypted at rest using encryption keys managed by Oracle. For example, kmsKeyId: ocid1.key.oc1.iad.ann______usj
      • nsgIds: '["<nsg-ocid>"]': Optional. A JSON array of up to five network security group OCIDs to associate with the Lustre file system. For example, nsgIds: '["ocid1.nsg.oc1.iad.aab______fea"]'
      • rootSquashEnabled: "<true | false>": Optional. Set to true to restrict root access from clients. Defaults to false.
      • rootSquashUid: "<value>": Optional. When root squash is enabled, root operations are mapped to this UID. Defaults to 65534.
      • rootSquashGid: "<value>": Optional. When root squash is enabled, root operations are mapped to this GID. Defaults to 65534.
      • rootSquashClientExceptions: '["<ip-address>"]': Optional. A JSON array of IP addresses or CIDR blocks that are not subject to root squash (maximum 10 entries). For example, rootSquashClientExceptions: '["10.0.2.4"]'.
      • oci.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}' Optional. Specifies a defined tag for the new file system. For example, oci.oraclecloud.com/initial-defined-tags-override: '{"Org": {"CostCenter": "AI"}}'

        Note that to apply defined tags from a tag namespace belonging to one compartment to a filesystem resource belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.

      • oci.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}' Optional. Specifies a free-form tag for the new file system. For example, oci.oraclecloud.com/initial-freeform-tags-override: '{"Project": "ML"}'
      • setupLnet: "<true | false>": Optional. Set to true if the Lustre CSI driver should perform Lustre Network (LNet) setup before mounting. We strongly recommend you include the setupLnet parameter and set it "true".
      • lustreSubnetCidr: "<cidr-block>": Optional. Set to the worker node's source network range used for Lustre traffic:
        • When to use: Only specify a network range if worker nodes use a secondary VNIC to connect to the Lustre file system. This CIDR must match the subnet block of that secondary VNIC (for example, 10.0.2.0/24).
        • When to omit: Do not specify a network range if worker nodes are using their primary VNIC (the default interface) for Lustre connectivity.
        • Important: This parameter is different to the Lustre file system's subnetId parameter, which defines where the Lustre file system itself is located.
      • lustrePostMountParameters: '[{"<parameter1>": <value>},{"<parameter2>": <value>}]': Optional. JSON array of advanced Lustre client parameters to set after mounting. For example, lustrePostMountParameters: '[{"*.*.*MDT*.lru_size": 11200},{"at_history": 600}]'

      For example:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: lustre-dyn-storage
      provisioner: lustre.csi.oraclecloud.com
      parameters:
        availabilityDomain: US-ASHBURN-AD-1
        compartmentId: ocid1.compartment.oc1..aaa______t6q # optional
        subnetId: ocid1.subnet.oc1.iad.aaaa______kfa
        performanceTier: MBPS_PER_TB_250
        fileSystemName: aiworkfs                           # optional
        kmsKeyId: ocid1.key.oc1.iad.ann______usj           # optional
        nsgIds: '["ocid1.nsg.oc1.iad.aab______fea"]'       # optional
        oci.oraclecloud.com/initial-defined-tags-override: '{"Org": {"CostCenter": "AI"}}'
        oci.oraclecloud.com/initial-freeform-tags-override: '{"Project": "ML"}'
        setupLnet: "true"                    # optional

   2. Create the storage class from the manifest file by entering:

      kubectl create -f <filename>

      For example:

      kubectl create -f lustre-dyn-st-class.yaml

3. Create a PVC to be provisioned by the new file system in the File Storage with Lustre service, as follows:
   1. Create a manifest file to define the PVC:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: <pvc-name>
      spec:
        accessModes:
          - <ReadWriteMany|ReadOnlyOncePod>
        storageClassName: "<storage-class-name>"
        resources:
          requests:
            storage: <capacity>

      where:

      • name: <pvc-name>: Required. For example, lustre-dynamic-claim
      • storageClassName: "<storage-class-name>": Required. The name of the storage class you defined earlier. For example, lustre-dyn-storage.
      • accessModes: - <ReadWriteMany|ReadOnlyOncePod>: Required. Specifies how the file system is to be mounted and shared by pods. Currently ReadWriteMany and ReadOnlyOncePod are supported. For example, ReadWriteMany.
      • storage: <capacity>: Required. This value must be at least 31.2T (or 31200G). You can specify a larger capacity, but you must use particular increments that depend on capacity (see Increasing File System Capacity). For example, 31.2T.

      For example, the following manifest file (named lustre-dyn-claim.yaml) defines a PVC named lustre-dynamic-claim that is provisioned by the file system defined in the lustre-dyn-storage storage class:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: lustre-dynamic-claim
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: "lustre-dyn-storage"
        resources:
          requests:
            storage: 31.2T

   2. Create the PVC from the manifest file by entering:

      kubectl create -f <filename> 

      For example:

      kubectl create -f lustre-dyn-claim.yaml

   A new PVC is created. The CSI volume plugin creates a new persistent volume (PV) and a new file system in the File Storage with Lustre service. Note that creating a new Lustre file system takes at least 10 minutes and can take longer, depending on the size of the file system. Use the Console or the CLI to confirm that the new Lustre file system has been created (see Listing File Systems).

   The new PVC is bound to the new PV. Data is encrypted at rest, using encryption keys managed either by Oracle or by you.

4. Verify that the PVC has been bound to the new persistent volume by entering:

   kubectl get pvc

   Example output:

   			
   NAME                   STATUS    VOLUME                    CAPACITY         ACCESSMODES   STORAGECLASS         AGE
   lustre-dynamic-claim   Bound     csi-lustre-<unique_ID>    30468750000Ki    RWX           lustre-dyn-storage   4m

5. Use the new PVC when creating other objects, such as pods. For example, you could create a new pod from the following pod definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: lustre-dynamic-app
   spec:
     containers:
       - name: aiworkload
         image: busybox:latest
         command: ["sleep", "3600"]
         volumeMounts:
           - name: lustre-vol
             mountPath: /mnt/lustre
     volumes:
       - name: lustre-vol
         persistentVolumeClaim:
           claimName: lustre-dynamic-claim

6. Having created a new pod as described in the example in the previous step, you can verify that the pod is using the new PVC by entering:

   kubectl describe pod lustre-dynamic-app

To create a PVC on an existing file system in the File Storage with Lustre service (using Oracle-managed encryption keys to encrypt data at rest):

1. Create a file system in the File Storage with Lustre service, selecting the Encrypt using Oracle-managed keys encryption option. See Creating a Lustre File System.
2. Create security rules in either a network security group (recommended) or a security list for both the Lustre file system, and for the cluster's worker nodes subnet.
   The security rules to create depend on the relative network locations of the Lustre file system and the worker nodes which act as the client, according to the following scenarios:

   • Option 1: Client and Lustre in Different Subnets
   • Option 2: Client and Lustre in the Same Subnet

   These scenarios, the security rules to create, and where to create them, are fully described in the File Storage with Lustre service documentation (see Required VCN Security Rules).

3. Create a PV backed by the file system in the File Storage with Lustre service as follows:

   1. Create a manifest file to define a PV and in the csi: section, set:

      • driver to lustre.csi.oraclecloud.com
      • volumeHandle to <MGSAddress>@<LNetName>:/<MountName>

        where:

        • <MGSAddress> is the Management service address for the file system in the File Storage with Lustre service
        • <LNetName> is the LNet network name for the file system in the File Storage with Lustre service.
        • <MountName> is the mount name used while creating the file system in the File Storage with Lustre service.

        For example: 10.0.2.6@tcp:/testlustrefs

      • fsType to lustre
      • (optional, but recommended) volumeAttributes.setupLnet to "true" if you want the Lustre CSI driver to perform lnet (Lustre Network) setup before mounting the filesystem.
      • (optional) volumeAttributes.lustreSubnetCidr to the worker node's source network range used for Lustre traffic:
        • When to use: Only specify a network range if worker nodes use a secondary VNIC to connect to the Lustre file system. This CIDR must match the subnet block of that secondary VNIC (for example, 10.0.2.0/24).
        • When to omit: Do not specify a network range if worker nodes are using their primary VNIC (the default interface) for Lustre connectivity.
        • Important: This parameter is different to the Lustre file system's subnetId parameter, which defines where the Lustre file system itself is located.
      • (optional) volumeAttributes.lustrePostMountParameters to set Lustre parameters. For example:

        ...
            volumeAttributes:
              lustrePostMountParameters: '[{"*.*.*MDT*.lru_size": 11200},{"at_history" :
            600}]'

      For example, the following manifest file (named lustre-pv-example.yaml) defines a PV called lustre-pv-example backed by a Lustre file system:

      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: lustre-pv-example
      spec:
        capacity:
          storage: 31.2T
        volumeMode: Filesystem
        accessModes:
          - ReadWriteMany
        persistentVolumeReclaimPolicy: Retain
        csi:
          driver: lustre.csi.oraclecloud.com
          volumeHandle: "10.0.2.6@tcp:/testlustrefs"
          fsType: lustre
          volumeAttributes:
            setupLnet: "true"

   2. Create the PV from the manifest file by entering:

      kubectl apply -f <filename>

      For example:

      kubectl apply -f lustre-pv-example.yaml

   3. Verify that the PV has been created successfully by entering:

      kubectl get pv <pv-name>

      For example:

      kubectl get pv lustre-pv-example

      Example output:

      
      NAME                CAPACITY        ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM   STORAGECLASS   REASON   AGE
      lustre-pv-example   30468750000Ki   RWX            Retain           Bound                                    56m
      

4. Create a PVC that is provisioned by the PV you have created, as follows:
   1. Create a manifest file to define the PVC and set:
      • storageClassName to "" Note that you must specify an empty value for storageClassName, even though storage class is not applicable in the case of static provisioning of persistent storage. If you do not specify an empty value for storageClassName, the default storage class (oci-bv) is used, which causes an error.
      • volumeName to the name of the PV you created (for example, lustre-pv-example)

      For example, the following manifest file (named lustre-pvc-example.yaml) defines a PVC named lustre-pvc-example that will bind to a PV named lustre-pv-example:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: lustre-pvc-example
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: ""
        volumeName: lustre-pv-example
        resources:
          requests:
            storage:  31.2T

      Note that the requests: storage: element must be present in the PVC's manifest file, and its value must match the value specified for the capacity: storage: element in the PV's manifest file. Apart from that, the value of the requests: storage: element is ignored.

   2. Create the PVC from the manifest file by entering:

      kubectl apply -f <filename>

      For example:

      kubectl apply -f lustre-pvc-example.yaml

   3. Verify that the PVC has been created and bound to the PV successfully by entering:

      kubectl get pvc <pvc-name>

      For example:

      kubectl get pvc lustre-pvc-example

      Example output:

      
      NAME                    STATUS   VOLUME              CAPACITY         ACCESS MODES   STORAGECLASS   AGE
      lustre-pvc-example      Bound    lustre-pv-example   30468750000Ki    RWX                           57m

   The PVC is bound to the PV backed by the File Storage with Lustre service file system. Data is encrypted at rest, using encryption keys managed by Oracle.

5. Use the new PVC when creating other objects, such as deployments. For example:
   1. Create a manifest named lustre-app-example-deployment.yaml to define a deployment named lustre-app-example-deployment that uses the lustre-pvc-example PVC, as follows:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: lustre-app-example-deployment
      spec:
        selector:
          matchLabels:
            app: lustre-app-example
        replicas: 2
        template:
          metadata:
            labels:
              app: lustre-app-example
          spec:
            containers:
            - args:
              - -c
              - while true; do echo $(date -u) >> /lustre/data/out.txt; sleep 60; done
              command:
              - /bin/sh
              image: busybox:latest
              imagePullPolicy: Always
              name: lustre-app-example
              volumeMounts:
              - mountPath: /lustre/data
                name: lustre-volume
            restartPolicy: Always
            volumes:
            - name: lustre-volume
              persistentVolumeClaim:
                claimName: lustre-pvc-example

   2. Create the deployment from the manifest file by entering:

      kubectl apply -f lustre-app-example-deployment.yaml

   3. Verify that the deployment pods have been created successfully and are running by entering:

      kubectl get pods

      Example output:

      NAME                                             READY   STATUS              RESTARTS   AGE
      lustre-app-example-deployment-7767fdff86-nd75n   1/1     Running             0          8h
      lustre-app-example-deployment-7767fdff86-wmxlh   1/1     Running             0          8h