Overview #
I’m using the following Debian 12 based nodes in this tutorial:
192.168.30.20 # Controller Node
192.168.30.21 # Worker Node
192.168.30.22 # Worker Node
192.168.30.23 # Worker Node
Create K3s Cluster #
Install Prerequisites #
Install the following dependencies on the controller and the worker nodes:
# Install prerequisites
sudo apt update && sudo apt upgrade -y &&
sudo apt install -y curl iptables jq
List Available K3s Versions #
Available K3s versions: https://github.com/k3s-io/k3s/tags
# List available K3s versions: Stable only
curl -s https://api.github.com/repos/k3s-io/k3s/releases | jq -r '.[] | select(.prerelease == false) | .tag_name' | grep 'k3s' | sort -V
# Shell output:
v1.29.13+k3s1
v1.29.14+k3s1
v1.30.9+k3s1
v1.30.10+k3s1
v1.31.4+k3s1
v1.31.5+k3s1
v1.31.6+k3s1
v1.32.0+k3s1
v1.32.1+k3s1
v1.32.2+k3s1
-
v1.32.2Kubernetes upstream version that K3s is based on -
+k3s1K3s-specific build, including optimizations and customizations
Install Controller Node #
Install Controller Node #
Install and enable K3s on the controller node:
# Install K3s: With Traefik Ingress
export K3S_VERSION="v1.30.10+k3s1"
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" sh -
Adapt Permissions #
The following steps create a new group called “kubectl” to allows other users then root to controll the cluster via Kubectl:
# Create a kubectl group
sudo groupadd kubectl
# Add the current user to the group
sudo usermod -aG kubectl $USER
# Change the group owner of the K3s config file
sudo chown root:kubectl /etc/rancher/k3s/k3s.yaml
# Adapt the permissions
sudo chmod 640 /etc/rancher/k3s/k3s.yaml
# Verify permissions
ls -l /etc/rancher/k3s/k3s.yaml
# Shell output:
-rw-r----- 1 root kubectl 2961 Mar 8 10:26 /etc/rancher/k3s/k3s.yaml
# Export env variable: Permanent
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
# Apply changes
source ~/.bashrc
Verify the K3s Cluster #
Verify the K3s cluster:
# List cluster nodes:
kubectl get nodes -o wide
# Shell output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-01 Ready control-plane,master 10m v1.30.10+k3s1 192.168.30.20 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
Copy the Token #
Get the token from the controller node that can be used to join the worker nodes to the cluster:
# Extract the token from the master node
sudo cat /var/lib/rancher/k3s/server/node-token
# Shell output:
K109dae8560dc61d3eb9e8113974b7d177d522bc257e375df13204621c7379cee75::server:ebb8279a2c0d4b91ed8c22e35a4e9b30
Install Worker Nodes #
Join Worker Nodes #
Run the following command on the worker node, to export the K3s version, Controller node token and IP as env variables:
# Add Worker Nodes
export K3S_VERSION="v1.30.10+k3s1"
export K3S_TOKEN="K109dae8560dc61d3eb9e8113974b7d177d522bc257e375df13204621c7379cee75::server:ebb8279a2c0d4b91ed8c22e35a4e9b30"
export SERVER_IP="192.168.30.20"
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" \
INSTALL_K3S_EXEC="agent" \
K3S_TOKEN="$K3S_TOKEN" \
K3S_URL="https://$SERVER_IP:6443" sh -
Label the Worker Nodes #
# Label the worker nodes
kubectl label nodes debian-02 kubernetes.io/role=worker &&
kubectl label nodes debian-03 kubernetes.io/role=worker &&
kubectl label nodes debian-04 kubernetes.io/role=worker
Verify the K3s cluster #
# List cluster nodes:
kubectl get nodes -o wide
# Shell output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-01 Ready control-plane,master 20m v1.30.10+k3s1 192.168.30.20 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-02 Ready worker 4m28s v1.30.10+k3s1 192.168.30.21 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-03 Ready worker 3m59s v1.30.10+k3s1 192.168.30.22 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-04 Ready worker 3m56s v1.30.10+k3s1 192.168.30.23 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
Taint the Controller Node #
Prevent regular workloads from being scheduled on the control plane node:
# Taint the master node
kubectl taint nodes debian-01 key=value:NoSchedule
# Verify node taints
kubectl describe node debian-01 | grep Taints
# Shell output:
Taints: key=value:NoSchedule
Install Helm #
# Install Helm with script
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&
chmod +x get_helm.sh &&
./get_helm.sh
Example Deployment #
Create an example deployment with NodePort service, so that the cluster runs a workload:
- example-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-deployment
labels:
app: example-deployment
spec:
replicas: 5
selector:
matchLabels:
app: example-container
template:
metadata:
labels:
app: example-container
spec:
containers:
- name: my-container
image: jueklu/container-2
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: my-container-service
spec:
type: NodePort
selector:
app: example-container
ports:
- protocol: TCP
port: 8080
targetPort: 8080
nodePort: 30080
# Apply the configuration
kubectl apply -f example-deployment.yaml
Verify the deployment:
# Verify the deployment
kubectl get pod -o wide
# Shell output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-deployment-747b9cbf6-69kqt 1/1 Running 0 47s 10.42.1.3 debian-02 <none> <none>
example-deployment-747b9cbf6-cb7d9 1/1 Running 0 47s 10.42.3.4 debian-04 <none> <none>
example-deployment-747b9cbf6-gmkzd 1/1 Running 0 47s 10.42.2.3 debian-03 <none> <none>
example-deployment-747b9cbf6-gmp25 1/1 Running 0 47s 10.42.1.4 debian-02 <none> <none>
example-deployment-747b9cbf6-lmv7w 1/1 Running 0 102s 10.42.3.3 debian-04 <none> <none>
Upgrade the K3s Cluster #
Upgrade the K3s cluster to a new version:
-
Upgrade the Controller node first
-
Drain, upgrade and uncordon each worker node in sequence
Backup SQLite DB #
# Install SQLite client
sudo apt install sqlite3 -y
# Verify the installation
sqlite3 --version
# Create a backup directory
sudo mkdir -p /k3s-backup
# Use SQLite dump to create a backup of the DB
sudo sqlite3 /var/lib/rancher/k3s/server/db/state.db ".backup '/k3s-backup/v1.30.10+k3s1_backup.db'"
Upgrade Controller Node #
# Stop the K3s service
systemctl stop k3s
# Upgrade the controller node
export K3S_VERSION="v1.31.6+k3s1"
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" sh -
# Start the K3s service
systemctl start k3s
# Verify the current version
k3s --version
# Shell output:
k3s version v1.31.6+k3s1 (6ab750f9)
go version go1.22.12
# Verify the cluster
kubectl get nodes -o wide
# Shell output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-01 Ready control-plane,master 51m v1.31.6+k3s1 192.168.30.20 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-02 Ready worker 34m v1.30.10+k3s1 192.168.30.21 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-03 Ready worker 34m v1.30.10+k3s1 192.168.30.22 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-04 Ready worker 34m v1.30.10+k3s1 192.168.30.23 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
# Verify the pods
kubectl get pods
# Shell output:
NAME READY STATUS RESTARTS AGE
example-deployment-747b9cbf6-69kqt 1/1 Running 0 5m48s
example-deployment-747b9cbf6-cb7d9 1/1 Running 0 5m48s
example-deployment-747b9cbf6-gmkzd 1/1 Running 0 5m48s
example-deployment-747b9cbf6-gmp25 1/1 Running 0 5m48s
example-deployment-747b9cbf6-lmv7w 1/1 Running 0 6m43s
Upgrade Worker Nodes: Node 1 #
# Drain the worker node
kubectl drain debian-02 --ignore-daemonsets
# Shell output:
Warning: ignoring DaemonSet-managed Pods: kube-system/svclb-traefik-55bec918-qxjcq
evicting pod default/example-deployment-747b9cbf6-gmp25
evicting pod default/example-deployment-747b9cbf6-69kqt
pod/example-deployment-747b9cbf6-69kqt evicted
pod/example-deployment-747b9cbf6-gmp25 evicted
node/debian-02 drained
# Upgrade the worker node
export K3S_VERSION="v1.31.6+k3s1"
export K3S_TOKEN="K109dae8560dc61d3eb9e8113974b7d177d522bc257e375df13204621c7379cee75::server:ebb8279a2c0d4b91ed8c22e35a4e9b30"
export SERVER_IP="192.168.30.20"
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" \
INSTALL_K3S_EXEC="agent" \
K3S_TOKEN="$K3S_TOKEN" \
K3S_URL="https://$SERVER_IP:6443" sh -
# Uncordon the worker node
kubectl uncordon debian-02
# Verify the cluster
kubectl get nodes -o wide
# Shell output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-01 Ready control-plane,master 51m v1.31.6+k3s1 192.168.30.20 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-02 Ready worker 34m v1.30.10+k3s1 192.168.30.21 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-03 Ready worker 34m v1.30.10+k3s1 192.168.30.22 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
debian-04 Ready worker 34m v1.30.10+k3s1 192.168.30.23 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
Upgrade Worker Nodes: Node 2 #
# Drain the worker node
kubectl drain debian-03 --ignore-daemonsets
# Upgrade the worker node
export K3S_VERSION="v1.31.6+k3s1"
export K3S_TOKEN="K109dae8560dc61d3eb9e8113974b7d177d522bc257e375df13204621c7379cee75::server:ebb8279a2c0d4b91ed8c22e35a4e9b30"
export SERVER_IP="192.168.30.20"
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" \
INSTALL_K3S_EXEC="agent" \
K3S_TOKEN="$K3S_TOKEN" \
K3S_URL="https://$SERVER_IP:6443" sh -
# Uncordon the worker node
kubectl uncordon debian-03
# Verify the cluster
kubectl get nodes -o wide
# Shell output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-01 Ready control-plane,master 64m v1.31.6+k3s1 192.168.30.20 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-02 Ready worker 47m v1.31.6+k3s1 192.168.30.21 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-03 Ready worker 47m v1.31.6+k3s1 192.168.30.22 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-04 Ready worker 47m v1.30.10+k3s1 192.168.30.23 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://1.7.23-k3s2
Upgrade Worker Nodes: Node 3 #
# Drain the worker node
kubectl drain debian-04 --ignore-daemonsets
# Upgrade the worker node
export K3S_VERSION="v1.31.6+k3s1"
export K3S_TOKEN="K109dae8560dc61d3eb9e8113974b7d177d522bc257e375df13204621c7379cee75::server:ebb8279a2c0d4b91ed8c22e35a4e9b30"
export SERVER_IP="192.168.30.20"
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" \
INSTALL_K3S_EXEC="agent" \
K3S_TOKEN="$K3S_TOKEN" \
K3S_URL="https://$SERVER_IP:6443" sh -
# Uncordon the worker node
kubectl uncordon debian-04
# Verify the cluster
kubectl get nodes -o wide
# Shell output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-01 Ready control-plane,master 65m v1.31.6+k3s1 192.168.30.20 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-02 Ready worker 49m v1.31.6+k3s1 192.168.30.21 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-03 Ready worker 48m v1.31.6+k3s1 192.168.30.22 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
debian-04 Ready worker 48m v1.31.6+k3s1 192.168.30.23 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-31-amd64 containerd://2.0.2-k3s2
# Verify the pods
kubectl get pods -o wide
# Shell output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-deployment-747b9cbf6-4rpnj 1/1 Running 0 3m53s 10.42.1.6 debian-02 <none> <none>
example-deployment-747b9cbf6-4znrw 1/1 Running 0 67s 10.42.2.7 debian-03 <none> <none>
example-deployment-747b9cbf6-64bc5 1/1 Running 0 3m53s 10.42.1.5 debian-02 <none> <none>
example-deployment-747b9cbf6-pck5s 1/1 Running 0 67s 10.42.2.8 debian-03 <none> <none>
example-deployment-747b9cbf6-rq75q 1/1 Running 0 67s 10.42.2.6 debian-03 <none> <none>