↓Skip to main content

Deploying with Flux: Deploy Manifests from private Repositories of a self-hosted GitLab Instance into Kubernetes Cluster

1526 words·
Flux GitLab Kubernetes K8s CoreDNS
Table of Contents
Flux - This article is part of a series.
Part 1: Deploying with Flux: Deploy Manifests from public and private GitHub Repositories into Kubernetes Cluster, Flux Commands
Part 2: Deploying with Flux: Deploy Manifests from public and private GitLab.com Repositories into Kubernetes Cluster
Part 3: This Article

Overview
#

I’m using a Kubeadm based K8s cluster with one controller and two worker nodes, with MetalLB and Nginx Ingress.

Make sure the Use Kubernetes Version 1.28, otherwise the latest Flux version (2.3.0) want work.

NAME      STATUS   ROLES           AGE     VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu1   Ready    control-plane   9m38s   v1.28.11   192.168.30.10   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18
ubuntu2   Ready    worker          7m19s   v1.28.11   192.168.30.11   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18
ubuntu3   Ready    worker          7m15s   v1.28.11   192.168.30.12   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18

Kubernetes Prerequisites
#

GitLab Hosts Entry
#

Make sure every Kubernetes node is able to resolve the GitLab DNS name:

# Add GitLab entries to /etc/hosts
sudo tee -a /etc/hosts <<EOF
192.168.70.4 gitlab.jklug.work
192.168.70.4 gitlab-registry.jklug.work
EOF

CoreDNS ConfigMap
#

Backup the ConfigMap
# 

# Export the current ConfigMap
kubectl get cm coredns -n kube-system -o yaml > coredns-configmap-backup.yaml

Add GitLab DNS Entry
#

Create a DNS entry that points to GitLab:

# Add the following DNS section to the CoreDNS ConfigMap
hosts {
    192.168.70.4 gitlab.jklug.work
    192.168.70.4 gitlab-registry.jklug.work
    fallthrough
}

# Edit the CoreDNS ConfigMap
kubectl edit cm coredns -n kube-system

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors {
        }
        health {
            lameduck 5s
        }
        ready
        kubernetes jkw-k8s.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts {
            192.168.70.4 gitlab.jklug.work
            192.168.70.4 gitlab-registry.jklug.work
            fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          prefer_udp
          max_concurrent 1000
        }
        cache 30

        loop
        reload
        loadbalance
    }    
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors {\n    }\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes jkw-k8s.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n      prefer_udp\n      max_concurrent 1000\n    }\n    cache 30\n\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2024-06-09T18:27:58Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: coredns
  namespace: kube-system
  resourceVersion: "116839"
  uid: 88b192f3-db32-42a4-a3d8-5dda86c0cfaf

# Restart CoreDNS
kubectl rollout restart deployment coredns -n kube-system

Verify DNS Resolution
# 

# Run a busybox pod
kubectl run busybox --image=busybox --restart=Never --stdin --tty

# Test the GitLab DNS resolution
nslookup gitlab.jklug.work

# Shell output:
Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   gitlab.jklug.work
Address: 192.168.70.4

# Exit the container terminal
exit

# Delete the busybox pod
kubectl delete pod busybox

GitLab
#

GitLab Access Token
#

Create a GitLab Access Token for Flux CLI:

  • Go to: (Avatar) > Edit profile > Access Tokens

  • Click “Add new token”

  • Define “Token name” Flux-Token

  • Select scopes: “API”

  • Click “Create personal access token”

# Copy the GitLab personal access tokens
glpat-khx54qyVgPoWnisLL9E3

Private Deployment Project
#

Create Project
#

I’m using my private gitlab.jklug.work repository named “k8s-example-deployment-private” to deploy it’s manifests into the Kubernetes cluster.

Repository URL: https://gitlab.jklug.work/root/k8s-example-deployment-private


Deployment Manifest
#

Create the following deployment manifest:

# Add the following file to the "k8s-example-deployment" repository:
deployments/nginx-deployment-private.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-private
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

The folderstructure looks like this

├── deployments
│   └── nginx-deployment-private.yaml
└── README.md

Flux
#

Install Flux CLI
#

Install Flux on a Controller node of the Kubernetes cluster:

# Install Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash

# Verify installation / check verison
flux --version

Verify Flux Compatibility
# 

# Check Kubernetes compatibility
flux check --pre

# Shell output:
► checking prerequisites
✔ Kubernetes 1.28.11 >=1.28.0-0
✔ prerequisites checks passed

Bootstrap Flux
# 

# Export GitLab token
export GITLAB_TOKEN=glpat-khx54qyVgPoWnisLL9E3

# Run the bootstrap command
flux bootstrap gitlab \
--hostname=gitlab.jklug.work:443 \
--owner=root \
--repository=flux-repo \
--branch=main \
--path=./clusters/my-cluster \
  --token-auth \
  --personal

# Shell output
► connecting to https://gitlab.jklug.work:443
✔ repository "https://gitlab.jklug.work:443/root/flux-repo" created
► cloning branch "main" from Git repository "https://gitlab.jklug.work:443/root/flux-repo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("23e862c5935d1716631d697c42b71ae069bb1e35")
► pushing component manifests to "https://gitlab.jklug.work:443/root/flux-repo.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("6c50a14b6b7dc1a763e1d0ee68abb39854260ba3")
► pushing sync manifests to "https://gitlab.jklug.work:443/root/flux-repo.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

Verify Flux Pods
#

Verify the Flux pods are up and running:

# List pods in "flux-system" namespace
kubectl get pods -n flux-system

# Shell output:
NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-5f7457c9dd-qxrpk           1/1     Running   0          35s
kustomize-controller-5f58d55f76-9x8rn      1/1     Running   0          35s
notification-controller-685bdc466d-pzrf9   1/1     Running   0          35s
source-controller-86b8b57796-qth9s         1/1     Running   0          35s

Check Flux Logs
# 

# Check the Flux logs
flux logs

# Shell output:
2024-07-08T11:07:15.777Z info Kustomization/flux-system.flux-system - Source artifact not found, retrying in 30s
2024-07-08T11:07:22.878Z info Kustomization/flux-system.flux-system - server-side apply for cluster definitions completed
2024-07-08T11:07:23.089Z info Kustomization/flux-system.flux-system - server-side apply completed
2024-07-08T11:07:23.104Z info Kustomization/flux-system.flux-system - Reconciliation finished in 862.133668ms, next run in 10m0s
2024-07-08T11:07:46.197Z info Kustomization/flux-system.flux-system - server-side apply for cluster definitions completed
2024-07-08T11:07:46.314Z info Kustomization/flux-system.flux-system - server-side apply completed
2024-07-08T11:07:46.327Z info Kustomization/flux-system.flux-system - Reconciliation finished in 543.113672ms, next run in 10m0s
2024-07-08T11:07:22.234Z info GitRepository/flux-system.flux-system - stored artifact for commit 'Add Flux sync manifests'

Flux Repository
#

Clone Repository
#

Clone the GitHub repository “flux-repo” created by Flux to a local host:

# Clone the Flux repository
git clone git@gitlab.jklug.work:root/flux-repo.git

The folder structure should look like this:

clusters
  └── my-cluster
      └── flux-system
          ├── gotk-components.yaml
          ├── gotk-sync.yaml
          └── kustomization.yaml

Install Flux CLI
#

Install the Flux CLI on the host where the Flux repository was cloned:

# Install Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash

Private Deployment Project
#

Create SSH Key for Flux
#

On the Kubernetes Controller node create a SSH key pair for Flux:

# Create a Deploy Key for GitLab
ssh-keygen -t rsa -b 4096 -f flux-deploy-key

# Copy the public SSH key
cat flux-deploy-key.pub

Add SSH Key to GitHub Repository
#

Add the SSH public key to the GitLab repository “k8s-example-deployment-private”.

  • Open the local GitLab repository

  • Go to (Project) “Settings” > “Repository”

  • Scroll down to “Deploy keys” & click “Expand”

  • Click “Add new key”

  • Add a title for the key like Flux-Key

  • Paste the public SSH key

# Public SSH key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCtLlSwio3W4uJCxOfiZS/QTm2u5ul+FEbt44SmoopzbTY1+mV5I/jnNAt7A1qOHPOD2NikQQPH6FAp1f4hyLbjOvgsQEjtnyq5/eDupIiAhg+Bmg5D9otORKS3tpqSJGxqSHgsiMGNmQlSqhomAZD9cvm3GZceiwuj333Qe38vwV4t2hIW++VH3a58lksGDItbnot2ti4O70BkGwYeRK69lzIOrLzXy/yb9M9V6NvhQWbb73GY5ArScVsPTQQW7hT7QdmZ7IDx+zVt7P038T3RCQRCNVOLPZW1Pn/u5mijjixGTEy0xSfyMYbqmcGdzNsrhEbAEOYc//g7QC5fcSMOxQGeGf4DdWyRhK52RRymGWWB5vcJhmtb0sBbSbUXH02LqeHU5TiAdn32SduYQwi7Z26k0coz80+pn27xE1DmZpWsWxk2UafxEwA53bGGrPE0uyX8VrFfNt0U4blWwMGu0FUjpWSC//9SDSRr7HABo849j2tgZ9PkRFi9ofTC7MyAQpwC/eIOIjk0Lty6F4pD1f/gGkhBXjLThihgPwuOl8IrUeeWgXE25YP9ZLSUJZc5TNACgOzIQ8RWGZ3s4xn0ZaYcmXYPUTeedo9XZpIfPOFWFbpwZ23FvfgZeOlN5yy/bUL32901AXNAC9NDUOlCtC+fh/YnLp2H0P1qyFRl2Q== ubuntu@ubuntu1
  • Click “Add key”

Kubernetes Secret
#

Create a Kubernetes secret with the Flux SSH private key. The following command also adds the host key for the GitLab repository:

# Create Kubernetes secret (Run command in same dir as the SSH key)
flux create secret git flux-git-deploy \
  --url=ssh://git@gitlab.jklug.work/root/k8s-example-deployment-private \
  --private-key-file=./flux-deploy-key \
  --namespace=flux-system

# Delete Kubernetes secret
kubectl delete secret flux-git-deploy -n flux-system

# Optional: Verify / decode the Kubernetes secret
kubectl get secret flux-git-deploy -n flux-system -o jsonpath="{.data.identity}" | base64 --decode

Add Repository Source
#

Define the GitHub example project “k8s-example-deployment” as source:

# Create repository source
flux create source git k8s-example-deployment-private \
  --url=ssh://git@gitlab.jklug.work/root/k8s-example-deployment-private \
  --branch=main \
  --interval=1m \
  --secret-ref=flux-git-deploy \
  --export > clusters/my-cluster/k8s-example-deployment-private-source.yaml

Add Repository Kustomization
#

This will configure Flux to build and apply the manifests in the “deployments” directory:

# Create a kustomization
flux create kustomization k8s-example-deployment-private \
  --target-namespace=default \
  --source=k8s-example-deployment-private \
  --path="./deployments" \
  --prune=true \
  --wait=true \
  --interval=30m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/my-cluster/k8s-example-deployment-private-kustomization.yaml

Verify Folder Structure
#

The folder structure should now look like this

# Flux folder structure
clusters
  └── my-cluster
      ├── flux-system
      │   ├── gotk-components.yaml
      │   ├── gotk-sync.yaml
      │   └── kustomization.yaml
      ├── k8s-example-deployment-private-kustomization.yaml
      └── k8s-example-deployment-private-source.yaml

Push
#

Push the kustomization and source manifests into the Flux GitHub repository:

# configure git
git config --global user.email "juergen@jklug.work"
git config --global user.name "Juergen"

Push repository source:

# Add source manifest and commit
git add clusters/my-cluster/k8s-example-deployment-private-source.yaml && git commit -m "Add Source"

# Push source to GitLab
git push

Verify Flux source:

# List Flux sources
flux get sources git

# Shell output:
NAME                            REVISION                SUSPENDED       READY   MESSAGE
flux-system                     main@sha1:029981fd      False           True    stored artifact for revision 'main@sha1:029981fd'
k8s-example-deployment-private  main@sha1:dc4c6f91      False           True    stored artifact for revision 'main@sha1:dc4c6f91'

Push repository kustomization:

# Add kustomization manifest and commit
git add clusters/my-cluster/k8s-example-deployment-private-kustomization.yaml && git commit -m "Add Kustomization"

# Push kustomization to GitLab
git push

Verify Flux kustomization:

# List Flux kustomizations
flux get kustomizations

# Shell output: Wait till ready
NAME                            REVISION                SUSPENDED       READY   MESSAGE
flux-system                     main@sha1:afdc5556      False           True    Applied revision: main@sha1:afdc5556
k8s-example-deployment-private  main@sha1:dc4c6f91      False           True    Applied revision: main@sha1:dc4c6f91

Verify the Deployment
# 

# List pods in the default namespace
kubectl get pods

# Shell output:
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-deployment-private-7c79c4bf97-dbzww   1/1     Running   0          72s
nginx-deployment-private-7c79c4bf97-gwjcq   1/1     Running   0          72s
nginx-deployment-private-7c79c4bf97-rwbcq   1/1     Running   0          72s

Stop the Deployment
# 

# Delete kustomization
flux delete kustomization k8s-example-deployment-private

# Delete source
flux delete source git k8s-example-deployment-private

# Verify the deletion: List all Sources & Kustomizations
flux get all

# Shell output:
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:afdc5556      False           True    stored artifact for revision 'main@sha1:afdc5556'

NAME                            REVISION                SUSPENDED       READY   MESSAGE
kustomization/flux-system       main@sha1:afdc5556      False           True    Applied revision: main@sha1:afdc5556

# Remove the source and kustomization from the repository
rm clusters/my-cluster/k8s-example-deployment-private-kustomization.yaml &&
rm clusters/my-cluster/k8s-example-deployment-private-source.yaml

# Remove the manifests and commit
git add -A && git commit -m "Remove Source and Kustomization"

# Push changes to GitLab
git push
Flux - This article is part of a series.
Part 1: Deploying with Flux: Deploy Manifests from public and private GitHub Repositories into Kubernetes Cluster, Flux Commands
Part 2: Deploying with Flux: Deploy Manifests from public and private GitLab.com Repositories into Kubernetes Cluster
Part 3: This Article