Container Storage Interface is a standardized interface in Kubernetes for exposing storage systems to containerized workloads.
Overview #
In this tutorial I’m using a K3s Kubernetes version 1.30 cluster and an Ubuntu 24.04 based NFS share, with the following details:
192.168.30.15 # Single node K3s Cluster
192.168.30.16 # NFS Server
# Kubernetes Cluster Details
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ubuntu6 Ready control-plane,master 91s v1.30.3+k3s1 192.168.30.15 <none> Ubuntu 24.04 LTS 6.8.0-40-generic containerd://1.7.17-k3s1
NFS CSI Driver #
NFS Server Setup #
Setup an NFS Server and share:
# Create directory for NFS share
sudo mkdir -p /srv/nfs/k8s_nfs-csi
# Install NFS package
sudo apt install nfs-kernel-server -y
# Open NFS configuration
sudo vi /etc/exports
# NFS configuration: Define the kubernetes nodes
/srv/nfs/k8s_nfs-csi 192.168.30.15(rw,sync,no_root_squash)
# Restart NFS server
sudo systemctl restart nfs-server
Install NFS Client on Kubernetes Nodes #
Install the NFS utilities package on the Kubernetes node:
# Install NFS utilities package and rpcbind package
sudo apt install nfs-common rpcbind -y
Note: The “rpcbind” package is necessary for NFSv3, which relies on remote procedure calls (RPCs) for various operations.
Verify the NFS connectivity:
# Verify that the NFS server is correctly configured
/usr/sbin/showmount -e 192.168.30.16
# Shell output:
Export list for 192.168.30.16:
/srv/nfs/k8s_nfs-csi 192.168.30.15
NFS CSI Driver #
Install the NFS CSI driver in the Kubernetes cluster. I use Helm for the installation.
Add Helm Repository #
# Install Helm via script
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&
chmod +x get_helm.sh &&
./get_helm.sh
# Add Helm repository & update repository index
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts &&
helm repo update
Install CSI NFS Driver #
# Install the CSI NFS Driver
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system
# Uninstall the CSI NFS Driver
helm uninstall csi-driver-nfs -n kube-system
# Shell output:
NAME: csi-driver-nfs
LAST DEPLOYED: Tue Aug 20 19:51:25 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The CSI NFS Driver is getting deployed to your cluster.
To check CSI NFS Driver pods status, please run:
kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/instance=csi-driver-nfs" --watch
Verify CSI NFS Driver #
# List pods
kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/instance=csi-driver-nfs" --watch
# Shell output:
NAME READY STATUS RESTARTS AGE
csi-nfs-controller-74d48b4cd7-4s9kj 4/4 Running 0 81s
csi-nfs-node-2bnrx 3/3 Running 0 81s
Create Storage Class #
vi csi-nfs-storage-class.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-csi
provisioner: nfs.csi.k8s.io # NFS CSI Driver
parameters:
server: 192.168.30.16
share: /srv/nfs/k8s_nfs-csi
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
- nfsvers=3
# Create storage class
kubectl apply -f csi-nfs-storage-class.yml
reclaimPolicy:
-
DeleteDeletes the volume when the PVC is deleted -
RetainKeeps the volume when the PVC is deleted
Verify Storage Class #
# List StorageClasses
kubectl get storageclasses
# Shell output:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 20m
nfs-csi nfs.csi.k8s.io Delete Immediate false 4s
# List StorageClass details
kubectl describe storageclasses nfs-csi
# Shell output:
Name: nfs-csi
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"nfs-csi"},"mountOptions":["nfsvers=3"],"parameters":{"server":"192.168.30.16","share":"/srv/nfs/k8s_nfs-csi"},"provisioner":"nfs.csi.k8s.io","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
Provisioner: nfs.csi.k8s.io
Parameters: server=192.168.30.16,share=/srv/nfs/k8s_nfs-csi
AllowVolumeExpansion: <unset>
MountOptions:
nfsvers=3
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
Create Persistent Volume Claim #
This will dynamically create a PersistentVolume in the cluster which utilizes the NFS server.
vi csi-nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-nfs-1
spec:
accessModes:
- ReadWriteMany # Must match underlying storage
resources:
requests:
storage: 2Gi # Define PV storage size
storageClassName: nfs-csi # Define the nfs-csi StorageClass
# Create persistant volume claim
kubectl apply -f csi-nfs-pvc.yaml
Verify Persistant Volume Claim #
# List persistant volume claims
kubectl get persistentvolumeclaims
# Shell output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
pvc-nfs-1 Bound pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e 2Gi RWX nfs-csi <unset> 5s
# List PVS details
kubectl describe pvc pvc-nfs-1
# Shell output:
Name: pvc-nfs-1
Namespace: default
StorageClass: nfs-csi
Status: Bound
Volume: pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
volume.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWX
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 14s (x2 over 14s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Normal Provisioning 14s nfs.csi.k8s.io_ubuntu6_34ed9d33-fc4d-4ff1-b5e2-cedc180cb437 External provisioner is provisioning volume for claim "default/pvc-nfs-1"
Normal ProvisioningSucceeded 14s nfs.csi.k8s.io_ubuntu6_34ed9d33-fc4d-4ff1-b5e2-cedc180cb437 Successfully provisioned volume pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e
Example Deployment using the PVC #
vi nginx-pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: default
name: example-deployment
labels:
app: nginx-nfs
spec:
replicas: 2
selector:
matchLabels:
app: nginx-nfs
template:
metadata:
labels:
app: nginx-nfs
spec:
containers:
- name: nginx-nfs
image: nginx:1.16.1-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: persistent-storage
mountPath: "/mnt/nfs" # Mountpoint inside the container
readOnly: false
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: pvc-nfs-1 # Define PVC name
restartPolicy: Always
kubectl apply -f nginx-pvc.yaml
Verify Deployment #
# List pods
kubectl get pods -l app=nginx-nfs
# Shell output:
NAME READY STATUS RESTARTS AGE
example-deployment-6887b56f47-mbgkk 1/1 Running 0 42s
example-deployment-6887b56f47-rsnfg 1/1 Running 0 42s
Verify the PVC Mount #
Verify PVC Details #
# List PVS details
kubectl describe pvc pvc-nfs-1
# Shell output:
Name: pvc-nfs-1
Namespace: default
StorageClass: nfs-csi
Status: Bound
Volume: pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
volume.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWX
VolumeMode: Filesystem
Used By: example-deployment-6887b56f47-mbgkk # Verify Pods
example-deployment-6887b56f47-rsnfg # Verify Pods
Verify Mount in Pod #
# List pod details
kubectl describe pod example-deployment-6887b56f47-mbgkk
# Shell output:
...
Containers:
nginx-nfs:
Container ID: containerd://8ae9d2f714c7e368ccfb784da5e62ccb53301c05375f81b3fffe280f2ed7aa32
Image: nginx:1.16.1-alpine
Image ID: docker.io/library/nginx@sha256:5057451e461dda671da5e951019ddbff9d96a751fc7d548053523ca1f848c1ad
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 20 Aug 2024 20:36:07 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/mnt/nfs from persistent-storage (rw) # Verify PVC mount
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h9gxz (ro)
Verify & Test Volume #
# Create file "/mnt/nfs/file1" in nginx-nfs pod1
kubectl exec -it example-deployment-6887b56f47-mbgkk -- touch /mnt/nfs/file1
# List files in "/mnt/nfs/file1" directory of nginx-nfs pod2
kubectl exec -it example-deployment-6887b56f47-rsnfg -- ls /mnt/nfs
# Shell output:
file1
Verify Files on NFS Share #
# List files
ls -la /srv/nfs/k8s_nfs-csi/pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e/
# Shell output:
drwxr-xr-x 2 root root 4096 Aug 20 20:42 .
drwxr-xr-x 3 root root 4096 Aug 20 20:26 ..
-rw-r--r-- 1 root root 0 Aug 20 20:42 file1
Delete Resources #
# Delete deployment
kubectl delete -f nginx-pvc.yaml
# List PV
kubectl get pv
# Shell output
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e 2Gi RWX Delete Bound default/pvc-nfs-1 nfs-csi <unset> 24m
# List PVC
kubectl get pvc
# Shell output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
pvc-nfs-1 Bound pvc-b5da6149-f39e-4634-8ad6-2ba2be02e61e 2Gi RWX nfs-csi <unset> 24m
# Delete PVC
kubectl delete pvc pvc-nfs-1
# Verify the PV is deleted:
kubectl get pv
# Shell output:
No resources found
# Delete storage class
kubectl delete storageclass nfs-csi
Links #
# CSI NFS Driver Helm Installation
https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/charts/README.md