Skip to main content

Cluster API Hetzner (CAPH) - Part 1: Deploy Kubernetes Cluster on Hetzner Cloud, with Cluster API from a KIND Management Cluster

2181 words·
Kubernetes Kubernetes Cluster Cluster API Kubeadm Hetzner Cloud KIND clusterctl Kubectl
Table of Contents

My Setup
#

In this tutorial I’m using an Ubuntu 24.04.2 LTS (noble) server with Docker installed as base for my KIND management cluster.



Hetzner Cloud Project
#

Create Project
#

Login to the Hetzner cloud console and create a new project: https://console.hetzner.cloud/projects


Hetzner API Token
#

Generate a new API token for the project:

  • Go to: (Project) > “Security” > API tokens

  • Click “Generate API token”

  • Add “read and write” permissions"

  • Copy the API token, it should look like this: 4euNmosomeAPIkeyOOToVZ

Add Public SSH Key
#

# Create SSH key pair: RSA 4096 bit
ssh-keygen -t rsa -b 4096

# Copy the public key
cat .ssh/id_rsa.pub

Add a public SSH key to the project:

  • Go to: (Project) > “Security” > “SSH keys”

  • Click “Add SSH key” and add a public SSH key

  • Enable the flag “Set as default key” and copy the name of the key



Kubernetes Management Cluster (KIND)
#

Overview
#

Cluster API requires a management cluster from which the workload cluster on Hetzner Cloud is deployed.

In this tutorial, I’m using a Docker-based KIND cluster for simplicity, but for a production setup, a single-node K3s cluster or similar would be more appropriate.


Install KIND & clusterctl & kubectl
#

Latest KIND version:
https://kind.sigs.k8s.io/docs/user/quick-start

Latest clusterctl version:
https://github.com/kubernetes-sigs/cluster-api/releases/latest


# Install KIND
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-amd64 &&
chmod +x kind &&
sudo mv kind /usr/local/bin/kind

# Install ClusterAPI CLI (clusterctl)
wget https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.10.1/clusterctl-linux-amd64
chmod +x clusterctl-linux-amd64
sudo mv clusterctl-linux-amd64 /usr/local/bin/clusterctl

# Install Kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&
chmod +x kubectl &&
sudo mv kubectl /usr/local/bin/

Verify the installation:

# Veirfy installation / list version
kind --version
clusterctl version
kubectl version --client

Create KIND Management Cluster
#

# Create a new KIND cluster
kind create cluster --name kind-caph-mgt

# Shell output:
Creating cluster "kind-caph-mgt" ...
 ✓ Ensuring node image (kindest/node:v1.32.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind-caph-mgt"
You can now use your cluster with:

kubectl cluster-info --context kind-kind-caph-mgt

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

Verify the KIND Cluster
#

# List KIND clusters
kind get clusters

# Shell output:
kind-caph-mgt
# List KIND cluster nodes
kubectl get nodes

# Shell output:
NAME                          STATUS   ROLES           AGE   VERSION
kind-caph-mgt-control-plane   Ready    control-plane   54s   v1.32.2

Initialize Clusterctl
#

# Initialize KIND cluster: Install the necessary Cluster API components
clusterctl init \
  --core cluster-api \
  --bootstrap kubeadm \
  --control-plane kubeadm \
  --infrastructure hetzner

# Shell output:
Fetching providers
Installing cert-manager version="v1.17.1"
Waiting for cert-manager to be available...
Installing provider="cluster-api" version="v1.10.1" targetNamespace="capi-system"
Installing provider="bootstrap-kubeadm" version="v1.10.1" targetNamespace="capi-kubeadm-bootstrap-system"
Installing provider="control-plane-kubeadm" version="v1.10.1" targetNamespace="capi-kubeadm-control-plane-system"
Installing provider="infrastructure-hetzner" version="v1.0.3" targetNamespace="caph-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Hetzner Project API Token
#

Add API Token as Kubernetes Secret
#

Add the previously created Hetzner project API token as Kubernetes secret:

# Create Kubernetes Secret
kubectl create secret generic hetzner --from-literal=hcloud=4euNmosomeAPIkeyOOToVZ

# Shell output:
secret/hetzner created
# Patch the created secret so it is automatically moved to the target cluster later
kubectl patch secret hetzner -p '{"metadata":{"labels":{"clusterctl.cluster.x-k8s.io/move":""}}}'

# Shell output:
secret/hetzner patched

Verify the Kubernetes Secret
#

# List secrets
kubectl get secret

# Shell output:
NAME      TYPE     DATA   AGE
hetzner   Opaque   1      48s



CAPH Managed Kubernetes Cluster: Deployment
#

Create Cluster Manifest
#

Kubernes Releases: https://kubernetes.io/releases/

export SSH_KEY_NAME="ubuntu@dockerplygrd"
export CLUSTER_NAME="jkw-caph"
export HCLOUD_REGION="fsn1"
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=2
export KUBERNETES_VERSION=1.31.6
export HCLOUD_CONTROL_PLANE_MACHINE_TYPE=cpx31
export HCLOUD_WORKER_MACHINE_TYPE=cpx31


# Generate CAPH cluster manifest
clusterctl generate cluster --infrastructure hetzner:v1.0.1 jkw-caph > jkw-caph-cluster.yaml
  • fsn1 Falkenstein Germany

  • cx31 2 vCPU, 8GB RAM, 80GB SSD

  • export SSH_KEY_NAME=ubuntu@dockerplygrd Must match the SSH key name in the Hetzner project


Deploy Workload Cluster
#

Deploy a Cluster API-managed Kubernetes (workload) cluster:

# Apply the workload cluster
kubectl apply -f jkw-caph-cluster.yaml

# Shell output:
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/jkw-caph-md-0 created
cluster.cluster.x-k8s.io/jkw-caph created
machinedeployment.cluster.x-k8s.io/jkw-caph-md-0 created
machinehealthcheck.cluster.x-k8s.io/jkw-caph-control-plane-unhealthy-5m created
machinehealthcheck.cluster.x-k8s.io/jkw-caph-md-0-unhealthy-5m created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/jkw-caph-control-plane created
hcloudmachinetemplate.infrastructure.cluster.x-k8s.io/jkw-caph-control-plane created
hcloudmachinetemplate.infrastructure.cluster.x-k8s.io/jkw-caph-md-0 created
hcloudremediationtemplate.infrastructure.cluster.x-k8s.io/control-plane-remediation-request created
hcloudremediationtemplate.infrastructure.cluster.x-k8s.io/worker-remediation-request created
hetznercluster.infrastructure.cluster.x-k8s.io/jkw-caph created

Verify the Workload Cluster
#

List Cluster Status
#

# List cluster status
kubectl get cluster jkw-caph

# Shell output:
NAME       CLUSTERCLASS   PHASE         AGE   VERSION
jkw-caph                  Provisioned   23s

List Control Planes
#

# List controlplanes
kubectl get kubeadmcontrolplanes

# Shell output:
NAME                     CLUSTER    INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE     VERSION
jkw-caph-control-plane   jkw-caph   true                                 1                  1         1             4m14s   v1.31.6

Note: The the controller node has the status “UNAVAILABLE” until the CNI is installed.


List Kubernetes Nodes
#

# List cluster nodes
kubectl get machines

# Shell output:
NAME                           CLUSTER    NODENAME   PROVIDERID          PHASE         AGE     VERSION
jkw-caph-control-plane-sw4h7   jkw-caph              hcloud://64140158   Provisioned   4m27s   v1.31.6
jkw-caph-md-0-s5frc-wd4ks      jkw-caph              hcloud://64140180   Provisioned   4m14s   v1.31.6
jkw-caph-md-0-s5frc-xlf6n      jkw-caph              hcloud://64140181   Provisioned   4m14s   v1.31.6

Check Logs
#

# Check the logs
kubectl get events --sort-by=.metadata.creationTimestamp -A | tail -30

Hetzner Cloud GUI
#

Verify that the VMs are created inside the Hetzner cloud project:

Verify that a LoadBalancer is created inside the Hetzner cloud project:

Note: If not otherwise defined, the load balancer is the type “LB11” by default.



CAPH Managed Kubernetes Cluster: Configuration
#

Kubeconfig Overview
#

Export Workload Cluster Kubeconfig
#

# Export the kubeconfig for the workload cluster
clusterctl get kubeconfig jkw-caph > kubeconfig-jkw-caph-cluster.yml

Switch Kubeconfig
#

# Use the Workload cluster kubeconfig
export KUBECONFIG=kubeconfig-jkw-caph-cluster.yml

# Use the KIND cluster kubeconfig
export KUBECONFIG=$HOME/.kube/config

Verify Current Kubeconfig
#

# Verify kubectl context
kubectl config current-context

# Shell output:
jkw-caph-admin@jkw-caph



Check Cluster Status
#

# List cluster nodes
kubectl get nodes -o wide

# Shell output:
NAME                           STATUS     ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
jkw-caph-control-plane-sw4h7   NotReady   control-plane   9m44s   v1.31.6   <none>        <none>        Ubuntu 22.04.5 LTS   5.15.0-138-generic   containerd://1.7.25
jkw-caph-md-0-s5frc-wd4ks      NotReady   <none>          7m42s   v1.31.6   <none>        <none>        Ubuntu 22.04.5 LTS   5.15.0-138-generic   containerd://1.7.25
jkw-caph-md-0-s5frc-xlf6n      NotReady   <none>          7m33s   v1.31.6   <none>        <none>        Ubuntu 22.04.5 LTS   5.15.0-138-generic   containerd://1.7.25

Note: The nodes are NotReady, because no CNI was installed yet.



CNI Plugin (Flannel)
#

Install Flannel CNI
#

# Install Flannel CNI
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Shell output:
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

Verify Flannel Pods
#

# Verify Flannel Pods
kubectl get pods -n kube-flannel

# Shell output:
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-cxqht   1/1     Running   0          9s
kube-flannel-ds-dmvnn   1/1     Running   0          9s
kube-flannel-ds-fgr58   1/1     Running   0          9s

Verify Cluster Status
#

After the CNI was installed, the Kubernetes nodes switch to status “Ready”:

# List cluster nodes
kubectl get nodes -o wide

# Shell output:
NAME                           STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
jkw-caph-control-plane-sw4h7   Ready    control-plane   11m     v1.31.6   <none>        <none>        Ubuntu 22.04.5 LTS   5.15.0-138-generic   containerd://1.7.25
jkw-caph-md-0-s5frc-wd4ks      Ready    <none>          9m6s    v1.31.6   <none>        <none>        Ubuntu 22.04.5 LTS   5.15.0-138-generic   containerd://1.7.25
jkw-caph-md-0-s5frc-xlf6n      Ready    <none>          8m57s   v1.31.6   <none>        <none>        Ubuntu 22.04.5 LTS   5.15.0-138-generic   containerd://1.7.25



Hetzner Cloud Controller (CCM)
#

Adapt CCM Configuration
#

CCM Manifest: https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/latest/download/ccm.yaml


Adapt the CCM configuration, so that it uses the correct Kubernetes secret with the Hetzner cloud project API token, that is used to allocate resources a LoadBalancer for the Ingress.

# Create a CCM configuration
vi hetzner-ccm.yml
---
# Source: hcloud-cloud-controller-manager/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hcloud-cloud-controller-manager
  namespace: kube-system
---
# Source: hcloud-cloud-controller-manager/templates/clusterrolebinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: "system:hcloud-cloud-controller-manager"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: hcloud-cloud-controller-manager
    namespace: kube-system
---
# Source: hcloud-cloud-controller-manager/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hcloud-cloud-controller-manager
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: hcloud-cloud-controller-manager
  template:
    metadata:
      labels:
        app: hcloud-cloud-controller-manager
    spec:
      serviceAccountName: hcloud-cloud-controller-manager
      dnsPolicy: Default
      tolerations:
        # Allow HCCM itself to schedule on nodes that have not yet been initialized by HCCM.
        - key: "node.cloudprovider.kubernetes.io/uninitialized"
          value: "true"
          effect: "NoSchedule"
        - key: "CriticalAddonsOnly"
          operator: "Exists"

        # Allow HCCM to schedule on control plane nodes.
        - key: "node-role.kubernetes.io/master"
          effect: NoSchedule
          operator: Exists
        - key: "node-role.kubernetes.io/control-plane"
          effect: NoSchedule
          operator: Exists

        - key: "node.kubernetes.io/not-ready"
          effect: "NoExecute"
      containers:
        - name: hcloud-cloud-controller-manager
          args:
            - "--allow-untagged-cloud"
            - "--cloud-provider=hcloud"
            - "--route-reconciliation-period=30s"
            - "--webhook-secure-port=0"
            - "--leader-elect=false"
          env:
            - name: HCLOUD_TOKEN
              valueFrom:
                secretKeyRef:
                  key: hcloud  # Adapt values for the Kubernetes secret
                  name: hetzner  # Adapt values for the Kubernetes secret
            - name: ROBOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: robot-password
                  name: hcloud
                  optional: true
            - name: ROBOT_USER
              valueFrom:
                secretKeyRef:
                  key: robot-user
                  name: hcloud
                  optional: true
          image: docker.io/hetznercloud/hcloud-cloud-controller-manager:v1.24.0 # x-releaser-pleaser-version
          ports:
            - name: metrics
              containerPort: 8233
          resources:
            requests:
              cpu: 100m
              memory: 50Mi
      priorityClassName: system-cluster-critical

Deploy Hetzer Cloud Controller
#

# Deploy Hetzer CCM
kubectl apply -f hetzner-ccm.yml

# Shell output:
serviceaccount/hcloud-cloud-controller-manager created
clusterrolebinding.rbac.authorization.k8s.io/system:hcloud-cloud-controller-manager created
deployment.apps/hcloud-cloud-controller-manager created

Verify Hetzner Cloud Controller
#

# Verify CCM pod
kubectl get pods -n kube-system

# Shell output:
NAME                                                   READY   STATUS    RESTARTS   AGE
coredns-7c65d6cfc9-pt9qt                               1/1     Running   0          15m
coredns-7c65d6cfc9-rrpdf                               1/1     Running   0          15m
etcd-jkw-caph-control-plane-sw4h7                      1/1     Running   0          16m
hcloud-cloud-controller-manager-b659ff755-wnchp        1/1     Running   0          65s # Check
kube-apiserver-jkw-caph-control-plane-sw4h7            1/1     Running   0          16m
kube-controller-manager-jkw-caph-control-plane-sw4h7   1/1     Running   0          16m
kube-proxy-tvtts                                       1/1     Running   0          15m
kube-scheduler-jkw-caph-control-plane-sw4h7            1/1     Running   0          16m



Install Helm
#

Make sure Helm is installed in the Workload cluster:

# Install Helm with script
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&
chmod +x get_helm.sh &&
./get_helm.sh

Install Nginx Ingress Controller
#

Add Helm Repository
#

# Add Helm chart
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# Update package index
helm repo update

Adapt Values
#

Nginx Ingress Values: https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml

# Create a configuration file for the Nginx Ingress Controller
vi nginx-ingress-hetzner-values.yml
controller:
  kind: DaemonSet
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

  service:
    enabled: true
    type: LoadBalancer
    annotations:
      load-balancer.hetzner.cloud/name: "k8s-ingress-lb"
      load-balancer.hetzner.cloud/location: "fsn1"  # or "hel1", "nbg1" etc.
      load-balancer.hetzner.cloud/type: "lb11"
      load-balancer.hetzner.cloud/ipv6-disabled: "true"
      load-balancer.hetzner.cloud/use-private-ip: "false"
      #load-balancer.hetzner.cloud/protocol: "https"
      # Replace below with your actual certificate name or ID if using TLS via Hetzner
      #load-balancer.hetzner.cloud/http-certificates: "your-certificate-name"
      #load-balancer.hetzner.cloud/http-redirect-http: "true"

    enableHttp: false

    targetPorts:
      https: https  # Forward LB port 443 to pod port 80 (unencrypted)

Install Ingress Controller
#

# Install the Nginx ingress controller 
helm install ingress-nginx ingress-nginx/ingress-nginx \
    --namespace ingress-nginx \
    --create-namespace \
    -f nginx-ingress-hetzner-values.yml

# Update Nginx Ingress controller
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  -n ingress-nginx \
  -f nginx-ingress-hetzner-values.yml

# Delete the Nginx Ingress controller
helm delete ingress-nginx \
    --namespace ingress-nginx
# Optional: Scale the Nginx Ingress 
kubectl scale deployment ingress-nginx-controller --replicas=3 -n ingress-nginx

Verify Nginx Ingress Controller
#

List Ingress Pods
#

# List pods
kubectl get pods -n ingress-nginx

# Shell outpout:
NAME                             READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-hlmqj   1/1     Running   0          110s
ingress-nginx-controller-nzd7s   1/1     Running   0          110s

List Ingress Class
#

# List IngressClass
kubectl get ingressclass

# Shell output:
NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       2m23s

List Ingress Service / Check External IP
#

Verify the Ingress service get’s an external IP from the Hetzner LoadBalancer:

# Verify Ingress service
kubectl get svc -n ingress-nginx

# Shell output:
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)         AGE
ingress-nginx-controller             LoadBalancer   10.105.99.240    138.199.128.104   443:30114/TCP   2m46s  # Check
ingress-nginx-controller-admission   ClusterIP      10.106.136.218   <none>            443/TCP         2m46s

Verify Ingress Controller Hetzner LoadBalancer
#

Verify that another LoadBalancer was created in the Hetzner Cloud project, that will be used by the Nginx Ingress Controller:



Example Deployment with Ingress Resource
#

Kubernetes Secret
#

In this tutorial I’m using a Let’s encrypt wildcard certificate, so that I don’t have to point an DNS entry to the LoadBalancer:

# Create a Kubernetes secret for the TLS certificate
kubectl create secret tls caph-example-tls --cert=./fullchain.pem --key=./privkey.pem

Deployment, ClusterIP Service & Ingress Resource
#

# Create a configuration for the example deployment
vi example-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jueklu-container-2
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jueklu-container-2
  template:
    metadata:
      labels:
        app: jueklu-container-2
    spec:
      containers:
      - name: jueklu-container-2
        image: jueklu/container-2
        ports:
        - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: jueklu-container-2
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jueklu-container-2

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jueklu-container-2-ingress
spec:
  ingressClassName: "nginx"
  tls:
  - hosts:
    - caph-example.jklug.work
    secretName: caph-example-tls
  rules:
  - host: caph-example.jklug.work
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: jueklu-container-2
            port:
              number: 8080
# Deploy the manifest
kubectl apply -f example-deployment.yaml

Verify the Deployment
#

List Pods
#

# List pods
kubectl get pods

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE
jueklu-container-2-5cc97ffb67-6nmnj   1/1     Running   0          31s
jueklu-container-2-5cc97ffb67-6vsg8   1/1     Running   0          31s
jueklu-container-2-5cc97ffb67-dq78d   1/1     Running   0          31s

Verify Ingress Resource / Check External IP
#

Verify the Ingress Resource get’s the external IP of the LoadBalancer:

# List the ingress resources
kubectl get ingress

# Shell output: (May takes a view minutes till the Ingress gets an external IP)
NAME                         CLASS   HOSTS                     ADDRESS           PORTS     AGE
jueklu-container-2-ingress   nginx   caph-example.jklug.work   138.199.128.104   80, 443   100s

DNS Entry
#

# Add a DNS entry for the Ingress Resource / LoadBalancer IP
138.199.128.104  caph-example.jklug.work

Access the Deployment
#

# Access the deployment with TLS encryption
https://caph-example.jklug.work
# Curl the deployment
curl https://caph-example.jklug.work

# Shell output: (The output should differ depending on the pod)
Container runs on: jueklu-container-2-5cc97ffb67-dq78d
Container runs on: jueklu-container-2-5cc97ffb67-6nmnj
Container runs on: jueklu-container-2-5cc97ffb67-6vsg8

Delete the Deployment
#

# Delete the deployment
kubectl delete -f example-deployment.yaml

# Delete the TLS secret
kubectl delete secret caph-example-tls



Delete Kubernetes Clusters
#

# Use the KIND cluster kubeconfig
export KUBECONFIG=$HOME/.kube/config

# Delete workload cluster
kubectl delete cluster jkw-caph

# Delete KIND cluster
kind delete cluster --name kind-caph-mgt



Links #

# Hetzner Cloud Locations
https://docs.hetzner.com/cloud/general/locations/

# Hetzner VM Prices
https://www.hetzner.com/cloud#pricing

# Hetzner LoadBalancer Prices
https://www.hetzner.com/de/cloud/load-balancer/
# CAPH Supported Kubernetes Versions
https://github.com/syself/cluster-api-provider-hetzner/blob/main/docs/caph/01-getting-started/01-introduction.md

# Kubernetes Releases:
https://kubernetes.io/releases/