Kubernetes Sets - StatefulSets: Difference between StatefulSet & ReplcaSet; StatefulSet Example with VolumeClaimTemplate and Stateless Service

# K3s Cluster
NAME      STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu1   Ready    control-plane,master   40d   v1.30.5+k3s1   192.168.30.10   <none>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.7.21-k3s2
ubuntu2   Ready    worker                 40d   v1.30.5+k3s1   192.168.30.11   <none>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.7.21-k3s2
ubuntu3   Ready    worker                 40d   v1.30.5+k3s1   192.168.30.12   <none>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.7.21-k3s2
ubuntu4   Ready    worker                 40d   v1.30.5+k3s1   192.168.30.13   <none>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.7.21-k3s2

192.168.30.90 # NFS Server / Ubuntu 22.04

CSI NFS StorageClass
#

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io # NFS CSI Driver
parameters:
  server: 192.168.30.90
  share: /srv/nfs/k8s_nfs-csi
reclaimPolicy: Retain
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=3

StatefulSet & ReplcaSet Overview
#

ReplicaSets (Deployments)
#

The ReplicaSet maintains the specified number of pod replicas running at all times.
When a deployment is created, Kubernetes automatically creates a ReplicaSet based on the settings in the deployment.
Used for stateless applications where each pod is identical, e.g. webservers.
Pods managed by a ReplicaSet are interchangeable and can be replaced with another without impacting the application.
When scaled, new pods are created with no concern for the order or individual pod identity.

StatefulSet
#

Manages pods that need unique, persistent storage across rescheduling.
Used for stateful applications where each pod needs to maintain its own data, e.g databases.
Each pod has a unique, stable identity, e.g. “pod-0”, “pod-1” and “pod-2” and keeps its identity even if rescheduled. If “pod-1” fails, it will be recreated with the same name, preserving its state.
StatefulSets ensure that Pods are created and deleted in a ordered manner.

Volumes in Deployments (ReplicaSets)
#

Each pod replica of the deployment shares the same PersistentVolumeClaim (PVC) which points to a single PersistentVolume (PV)
Each pod replica is typically interchangeable and does not require its own unique storage.

Volumes in StatefulSets
#

Typically a volumeClaimTemplate is specified for a StatefulSet. It’s used as a blueprint to dynamically provision a new PersistentVolume (PV) for each Pod.
The same PersistentVolume (PV) is reattached to the pod if it is rescheduled.

StatefulSet Example
#

VolumeClaimTemplate Overview
#

Example VolumeClaimTemplate:

volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage  # Name of the VolumeClaim for each pod
    spec:
      accessModes: 
        - ReadWriteOnce  # Allows only a single pod to mount the volume at a time
      resources:
        requests:
          storage: 1Gi  # 1 GB for each pods volume
      storageClassName: standard  # Define the storage class

Example PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongo-persistent-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

Example StatefulSet (MongoDB)
#

# Create a manifest for the StatefulSet
vi mongodb-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo-service"
  replicas: 3
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo:latest
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-persistent-storage
          mountPath: /data/db

  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: nfs-csi

# Create the StatefulSet
kubectl create -f mongodb-statefulset.yaml

spec: serviceName: "mongo": Refers to a headless service that is associated with the StatefulSet and enables each pod to be individually addressable.
This service does not load-balance traffic as a regular service does. Instead, it allows each pod in the StatefulSet to be accessed directly using a unique DNS entry.

Each pod in the StatefulSet will get a predictable hostname following the pattern:

# Pattern syntax
<pod-name>.<service-name>

# Pattern example
mongodb-0.mongo-service
mongodb-1.mongo-service
mongodb-2.mongo-service

Headless Service for StatefulSet
#

# Create a manifest for the service
vi mongodb-headless-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: mongo-service
  labels:
    app: mongo
spec:
  ports:
  - port: 27017
    name: mongo-service
  clusterIP: None  # Define headless service
  selector:
    app: mongo

# Create the service
kubectl create -f mongodb-headless-service.yaml

Verify Resources
#

Pods & StatefulSet
#

# List pods
kubectl get pods

# Shell output:
NAME                                    READY   STATUS    RESTARTS      AGE
mongo-0                                 1/1     Running   0             119s
mongo-1                                 1/1     Running   0             99s
mongo-2                                 1/1     Running   0             81s

Verify the StatefulSet:

# List StatefulSets
kubectl get statefulsets

# Shell output:
NAME                 READY   AGE
mongo                3/3     2m23s

PersistentVolumeClaims & PersistentVolumes:
#

# List PVCs
kubectl get pvc

# Shell output:
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
mongo-persistent-storage-mongo-0   Bound    pvc-21d55cf1-de64-4192-a204-c3c4c7b60640   1Gi        RWO            nfs-csi        <unset>                 4m14s
mongo-persistent-storage-mongo-1   Bound    pvc-4ddb3683-7af4-47db-ac0d-89f526493db3   1Gi        RWO            nfs-csi        <unset>                 3m54s
mongo-persistent-storage-mongo-2   Bound    pvc-96d49886-5c23-45ad-8f55-f1d87478ddf3   1Gi        RWO            nfs-csi        <unset>                 3m36s

# List PVCs
kubectl get pv

# Shell output:
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                      STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-21d55cf1-de64-4192-a204-c3c4c7b60640   1Gi        RWO            Retain           Bound    default/mongo-persistent-storage-mongo-0   nfs-csi        <unset>                          5m20s
pvc-4ddb3683-7af4-47db-ac0d-89f526493db3   1Gi        RWO            Retain           Bound    default/mongo-persistent-storage-mongo-1   nfs-csi        <unset>                          5m
pvc-96d49886-5c23-45ad-8f55-f1d87478ddf3   1Gi        RWO            Retain           Bound    default/mongo-persistent-storage-mongo-2   nfs-csi        <unset>                          4m42s

Stateless Service:
#

# List services
kubectl get svc

# Shell output:
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP                                               PORT(S)          AGE
mongo-service               ClusterIP      None            <none>                                                    27017/TCP        5s

Addressing Each Pod via Service
#

Verify DNS Names
#

Verify the DNS names:

# Start a Busybox pod
kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm

# Verify the DNS names
nslookup mongo-0.mongo-service
nslookup mongo-1.mongo-service
nslookup mongo-2.mongo-service


# Shell output:
/ # nslookup mongo-0.mongo-service
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mongo-0.mongo-service
Address 1: 10.42.2.12 mongo-0.mongo-service.default.svc.cluster.local

/ # nslookup mongo-1.mongo-service
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mongo-1.mongo-service
Address 1: 10.42.3.11 mongo-1.mongo-service.default.svc.cluster.local

/ # nslookup mongo-2.mongo-service
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mongo-2.mongo-service
Address 1: 10.42.1.9 mongo-2.mongo-service.default.svc.cluster.local

# Exit the busybox container
exit

The DNS names should look like this:

# DNS name syntax
<pod-name>.<service-name>.<namespace>.svc.cluster.local

# DNS name examples
mongo-0: mongo-0.mongo-service.default.svc.cluster.local
mongo-1: mongo-1.mongo-service.default.svc.cluster.local
mongo-2: mongo-2.mongo-service.default.svc.cluster.local

Connect to MongoDB via Serivce DNS Name
#

Run an Ubuntu pod:

# Run a Ubuntu pod
kubectl run -i --tty dns-test --image=ubuntu --restart=Never --rm -- bash

Install the MongoDB Shell:

# Install dependencies
apt update && apt install curl gpg sudo -y

# Import the MongoDB public GPG key
curl -fsSL https://www.mongodb.org/static/pgp/server-8.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-8.0.gpg \
   --dearmor

# Create the List File
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list

# Install the MongoDB Client
sudo apt update &&
sudo apt-get install -y mongodb-mongosh

# Verify the Installation / Check the version
mongosh --version

# Shell output:
2.3.3

Connect to a Mongo pod:

# Connect to a Mongo pod via service
mongosh --host mongo-0.mongo-service.default.svc.cluster.local --port 27017

# List databases
show dbs

# Shell output:
admin   40.00 KiB
config  60.00 KiB
local   40.00 KiB

# Exit the Mongo shell
exit

Exit the Ubuntu pod:

# Exit the Ubuntu pod:
exit

Delete the Resources
#

# Delete the StatefulSet and its pods
kubectl delete statefulset mongo

# Delete the PersistentVolumeClaims
kubectl delete pvc mongo-persistent-storage-mongo-0 mongo-persistent-storage-mongo-1 mongo-persistent-storage-mongo-2

# Delete the PersistentVolumes
kubectl delete pv pvc-21d55cf1-de64-4192-a204-c3c4c7b60640 pvc-4ddb3683-7af4-47db-ac0d-89f526493db3 pvc-96d49886-5c23-45ad-8f55-f1d87478ddf3

Links
#

# Install Mongo shell
https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition-using-deb-packages
https://www.mongodb.com/docs/mongodb-shell/install/

# Kubernetes Official Documentation: StatefulSet with Nginx Pod
https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

Kubernetes-Components - This article is part of a series.

Part 1: Kubernetes Configuration Manifests: Create Manifests from RAW Output and Dry-Run Command

Part 2: Kubernetes Non-Disruptive & Disruptive Configuration Updates: Kubectl Apply, Edit, Patch & Replace; Update Rollouts and Rollbacks with Set Image Command

Part 3: Kubernetes Pods: Create Pods with Run-Command and YAML Configuration; Single & Multi Container Pods, Port-Forwarding, Find Container on Worker Node

Part 4: Kubernetes Pods: Init & Sidecar Container Overview, Init Container Examples

Part 6: Kubernetes Services: Example ClusterIP, NodePort & LoadBalancer Services with Expose-Command and YAML Configuration; Service for External Endpoint

Part 8: Kubernetes Monitoring & Logs: Monitor Applications with top, Monitor Events Pod specific and Cluster wide, Container STDOUT and STDERR Logs

Part 9: Kubernetes Security: Immutable Deployment - Deploy Container with ReadOnly-Filesystem and Writable-Volume

Part 10: Kubernetes Security: Pod Security Admission (PSA) - Overview, Enforce Pod Security Standard at a Namespace; Example Nginx Pod SecurityContext for Restricted PSS

Part 12: Kubernetes Secrets: Opaque Secret Configuration, Pod Examples with Environment Variable Secrets and Volume Secrets; SSH Authentication Secret with Pod Example

Part 13: Kubernetes ConfigMaps: Mount ConfigMap to Pod as Volume, Mount ConfigMap as Environment Variable

Part 14: Kubernetes Sets - ReplicaSets & DaemonSets: Overview, Example ReplicaSet, Example DaemonSet with and without NodeSelector / Node Labeling