↓Skip to main content

Kubernetes Container Storage Interface (CSI): NFS CSI Driver Example, Dynamically create PersistentVolumes using a CSI StorageClass and PVC

1108 words·
Kubernetes CSI Kubectl Helm NFS
Table of Contents
Kubernetes-CSI - This article is part of a series.
Part 1: This Article
Part 2: Kubernetes Container Storage Interface (CSI): Ceph CSI Driver Example, Dynamically provision RBD Images in a Ceph Storage Pool
Part 3: Kubernetes Container Storage Interface (CSI): Longhorn Distributed Block Storage System - Deploy Longhorn via Helm Chart, Define Custom Storage Mountpoints; StorageClass & PVC Example
Part 4: Kubernetes Container Storage Interface (CSI): Samba - Deploy SMB CSI via Helm, Create example PVC and Pod

Container Storage Interface is a standardized interface in Kubernetes for exposing storage systems to containerized workloads.

Overview
#

In this tutorial I’m using a K3s Kubernetes version 1.30 cluster and an Ubuntu 24.04 based NFS share, with the following details:

192.168.30.15 # Single node K3s Cluster
192.168.30.16 # NFS Server

# Kubernetes Cluster Details
NAME      STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu6   Ready    control-plane,master   91s   v1.30.3+k3s1   192.168.30.15   <none>        Ubuntu 24.04 LTS   6.8.0-40-generic   containerd://1.7.17-k3s1



NFS CSI Driver
#

NFS Server Setup
#

Setup an NFS Server and share:

# Create directory for NFS share
sudo mkdir -p /srv/nfs/k8s_nfs-csi

# Install NFS package
sudo apt install nfs-kernel-server -y

# Open NFS configuration
sudo vi /etc/exports

# NFS configuration: Define the kubernetes nodes
/srv/nfs/k8s_nfs-csi 192.168.30.15(rw,sync,no_root_squash)

# Restart NFS server
sudo systemctl restart nfs-server

Install NFS Client on Kubernetes Nodes
#

Install the NFS utilities package on the Kubernetes node:

# Install NFS utilities package and rpcbind package
sudo apt install nfs-common rpcbind -y

Note: The “rpcbind” package is necessary for NFSv3, which relies on remote procedure calls (RPCs) for various operations.


Verify the NFS connectivity:

# Verify that the NFS server is correctly configured
/usr/sbin/showmount -e 192.168.30.16

# Shell output:
Export list for 192.168.30.16:
/srv/nfs/k8s_nfs-csi 192.168.30.15

NFS CSI Driver
#

Install the NFS CSI driver in the Kubernetes cluster. I use Helm for the installation.


Add Helm Repository
# 

# Install Helm via script
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&
chmod +x get_helm.sh &&
./get_helm.sh

# Add Helm repository & update repository index
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts &&
helm repo update

Install CSI NFS Driver
# 

# Install the CSI NFS Driver
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system

# Uninstall the CSI NFS Driver
helm uninstall csi-driver-nfs -n kube-system

# Shell output:
NAME: csi-driver-nfs
LAST DEPLOYED: Tue Aug 20 19:51:25 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The CSI NFS Driver is getting deployed to your cluster.

To check CSI NFS Driver pods status, please run:

  kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/instance=csi-driver-nfs" --watch

Verify CSI NFS Driver
# 

# List pods
kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/instance=csi-driver-nfs" --watch

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE
csi-nfs-controller-74d48b4cd7-4s9kj   4/4     Running   0          81s
csi-nfs-node-2bnrx                    3/3     Running   0          81s

Create Storage Class
# 

vi csi-nfs-storage-class.yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io # NFS CSI Driver
parameters:
  server: 192.168.30.16
  share: /srv/nfs/k8s_nfs-csi
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=3

# Create storage class
kubectl apply -f csi-nfs-storage-class.yml

reclaimPolicy:

  • Delete Deletes the volume when the PVC is deleted

  • Retain Keeps the volume when the PVC is deleted


Verify Storage Class
# 

# List StorageClasses
kubectl get storageclasses

# Shell output:
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  20m
nfs-csi                nfs.csi.k8s.io          Delete          Immediate              false                  4s

# List StorageClass details
kubectl describe storageclasses nfs-csi

# Shell output:
Name:            nfs-csi
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"nfs-csi"},"mountOptions":["nfsvers=3"],"parameters":{"server":"192.168.30.16","share":"/srv/nfs/k8s_nfs-csi"},"provisioner":"nfs.csi.k8s.io","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}

Provisioner:           nfs.csi.k8s.io
Parameters:            server=192.168.30.16,share=/srv/nfs/k8s_nfs-csi
AllowVolumeExpansion:  <unset>
MountOptions:
  nfsvers=3
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

Create Persistent Volume Claim
#

This will dynamically create a PersistentVolume in the cluster which utilizes the NFS server.

vi csi-nfs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-1
spec:
  accessModes:
    - ReadWriteMany # Must match underlying storage
  resources:
    requests:
      storage: 2Gi # Define PV storage size
  storageClassName: nfs-csi # Define the nfs-csi StorageClass

# Create persistant volume claim
kubectl apply -f csi-nfs-pvc.yaml

Verify Persistant Volume Claim
# 

# List persistant volume claims
kubectl get persistentvolumeclaims

# Shell output:
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-nfs-1   Bound    pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e   2Gi        RWX            nfs-csi        <unset>                 5s

# List PVS details
kubectl describe pvc pvc-nfs-1

# Shell output:
Name:          pvc-nfs-1
Namespace:     default
StorageClass:  nfs-csi
Status:        Bound
Volume:        pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
               volume.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                 Age                From                                                         Message
  ----    ------                 ----               ----                                                         -------
  Normal  ExternalProvisioning   14s (x2 over 14s)  persistentvolume-controller                                  Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  Provisioning           14s                nfs.csi.k8s.io_ubuntu6_34ed9d33-fc4d-4ff1-b5e2-cedc180cb437  External provisioner is provisioning volume for claim "default/pvc-nfs-1"
  Normal  ProvisioningSucceeded  14s                nfs.csi.k8s.io_ubuntu6_34ed9d33-fc4d-4ff1-b5e2-cedc180cb437  Successfully provisioned volume pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e

Example Deployment using the PVC
# 

vi nginx-pvc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: example-deployment
  labels:
    app: nginx-nfs
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-nfs
  template:
    metadata:
      labels:
        app: nginx-nfs
    spec:
      containers:
        - name: nginx-nfs
          image: nginx:1.16.1-alpine
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: persistent-storage
              mountPath: "/mnt/nfs" # Mountpoint inside the container
              readOnly: false
      volumes:
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: pvc-nfs-1  # Define PVC name
      restartPolicy: Always

kubectl apply -f nginx-pvc.yaml

Verify Deployment
# 

# List pods
kubectl get pods -l app=nginx-nfs

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE
example-deployment-6887b56f47-mbgkk   1/1     Running   0          42s
example-deployment-6887b56f47-rsnfg   1/1     Running   0          42s

Verify the PVC Mount
#

Verify PVC Details
# 

# List PVS details
kubectl describe pvc pvc-nfs-1

# Shell output:
Name:          pvc-nfs-1
Namespace:     default
StorageClass:  nfs-csi
Status:        Bound
Volume:        pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
               volume.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       example-deployment-6887b56f47-mbgkk # Verify Pods
               example-deployment-6887b56f47-rsnfg # Verify Pods

Verify Mount in Pod
# 

# List pod details
kubectl describe pod example-deployment-6887b56f47-mbgkk

# Shell output:
...
Containers:
  nginx-nfs:
    Container ID:   containerd://8ae9d2f714c7e368ccfb784da5e62ccb53301c05375f81b3fffe280f2ed7aa32
    Image:          nginx:1.16.1-alpine
    Image ID:       docker.io/library/nginx@sha256:5057451e461dda671da5e951019ddbff9d96a751fc7d548053523ca1f848c1ad
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 20 Aug 2024 20:36:07 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/nfs from persistent-storage (rw) # Verify PVC mount
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h9gxz (ro)

Verify & Test Volume
# 

# Create file "/mnt/nfs/file1" in nginx-nfs pod1
kubectl exec -it example-deployment-6887b56f47-mbgkk -- touch /mnt/nfs/file1

# List files in "/mnt/nfs/file1" directory of nginx-nfs pod2
kubectl exec -it example-deployment-6887b56f47-rsnfg -- ls /mnt/nfs

# Shell output:
file1

Verify Files on NFS Share
# 

# List files
ls -la /srv/nfs/k8s_nfs-csi/pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e/

# Shell output:
drwxr-xr-x 2 root root 4096 Aug 20 20:42 .
drwxr-xr-x 3 root root 4096 Aug 20 20:26 ..
-rw-r--r-- 1 root root    0 Aug 20 20:42 file1

Delete Resources
# 

# Delete deployment
kubectl delete -f nginx-pvc.yaml

# List PV
kubectl get pv

# Shell output
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e   2Gi        RWX            Delete           Bound    default/pvc-nfs-1   nfs-csi        <unset>                          24m


# List PVC
kubectl get pvc

# Shell output:
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-nfs-1   Bound    pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e   2Gi        RWX            nfs-csi        <unset>                 24m

# Delete PVC
kubectl delete pvc pvc-nfs-1

# Verify the PV is deleted:
kubectl get pv

# Shell output:
No resources found

# Delete storage class
kubectl delete storageclass nfs-csi

Links # 

# CSI NFS Driver Helm Installation
https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/charts/README.md
Kubernetes-CSI - This article is part of a series.
Part 1: This Article
Part 2: Kubernetes Container Storage Interface (CSI): Ceph CSI Driver Example, Dynamically provision RBD Images in a Ceph Storage Pool
Part 3: Kubernetes Container Storage Interface (CSI): Longhorn Distributed Block Storage System - Deploy Longhorn via Helm Chart, Define Custom Storage Mountpoints; StorageClass & PVC Example
Part 4: Kubernetes Container Storage Interface (CSI): Samba - Deploy SMB CSI via Helm, Create example PVC and Pod