↓Skip to main content

Deploying with Flux: Deploy Manifests from public and private GitLab.com Repositories into Kubernetes Cluster

1779 words·
Flux GitLab Kubernetes K8s
Table of Contents
Flux - This article is part of a series.
Part 1: Deploying with Flux: Deploy Manifests from public and private GitHub Repositories into Kubernetes Cluster, Flux Commands
Part 2: This Article
Part 3: Deploying with Flux: Deploy Manifests from private Repositories of a self-hosted GitLab Instance into Kubernetes Cluster

Overview
#

I’m using a Kubeadm based K8s cluster with one controller and two worker nodes, with MetalLB and Nginx Ingress.

Make sure the Use Kubernetes Version 1.28, otherwise the latest Flux version (2.3.0) want work.

NAME      STATUS   ROLES           AGE     VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu1   Ready    control-plane   9m38s   v1.28.11   192.168.30.10   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18
ubuntu2   Ready    worker          7m19s   v1.28.11   192.168.30.11   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18
ubuntu3   Ready    worker          7m15s   v1.28.11   192.168.30.12   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18

GitLab.com
#

GitLab Access Token
#

Create a GitLab Access Token for Flux CLI:

  • Go to: (Avatar) > Edit profile > Access Tokens

  • Click “Add new token”

  • Define a “Token name” like Flux-Token

  • Select scopes: “API”

  • Click “Create personal access token”

# Cope the GitLab API token
glpat-zGJ2b...

Public Deployment Project
#

Create Project
#

I’m using my public GitLab.com repository named “k8s-example-deployment” to deploy it’s manifests into the Kubernetes cluster.

Repository URL: https://gitlab.com/jueklug/k8s-example-deployment


Deployment Manifest
#

Create the following deployment manifest:

# Add the following file to the "k8s-example-deployment" repository:
deployments/nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

The folderstructure looks like this

├── deployments
│   └── nginx-deployment.yaml
└── README.md

Private Deployment Project
#

Create Project
#

I’m using my private GitLab.com repository named “k8s-example-deployment-private” to deploy it’s manifests into the Kubernetes cluster.

Repository URL: https://gitlab.com/jueklug/k8s-example-deployment-private


Deployment Manifest
#

Create the following deployment manifest:

# Add the following file to the "k8s-example-deployment" repository:
deployments/nginx-deployment-private.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-private
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

The folderstructure looks like this

├── deployments
│   └── nginx-deployment-private.yaml
└── README.md

Kubernetes Cluster
#

Install Flux CLI
# 

# Install Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash

# Verify installation / check verison
flux --version

Verify Flux Compatibility
# 

# Check Kubernetes compatibility
flux check --pre

# Shell output:
► checking prerequisites
✔ Kubernetes 1.28.11 >=1.28.0-0
✔ prerequisites checks passed

Bootstrap Flux
# 

# Export GitLab token
export GITLAB_TOKEN=glpat-zGJ2b...

# Run the bootstrap command
flux bootstrap gitlab \
  --hostname=gitlab.com \
  --owner=jueklug \
  --repository=flux-repo \
  --branch=main \
  --path=./clusters/my-cluster \
  --token-auth \
  --personal

# Shell output
► connecting to https://gitlab.com
✔ repository "https://gitlab.com/jueklug/flux-repo" created
► cloning branch "main" from Git repository "https://gitlab.com/jueklug/flux-repo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("6e9d136715c65fa6d24aa92dec3102138f")
► pushing component manifests to "https://gitlab.com/jueklug/flux-repo.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("2b01d2b88a6d7289d0df5beae2c15")
► pushing sync manifests to "https://gitlab.com/jueklug/flux-repo.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

Verify Flux Pods
#

Verify the Flux pods are up and running:

# List pods in "flux-system" namespace
kubectl get pods -n flux-system

# Shell output:
NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-5f7457c9dd-7m8bm           1/1     Running   0          98s
kustomize-controller-5f58d55f76-2kbs8      1/1     Running   0          98s
notification-controller-685bdc466d-xz48k   1/1     Running   0          98s
source-controller-86b8b57796-l92q7         1/1     Running   0          98s

Check Flux Logs
# 

# Check the Flux logs
flux logs

# Shell output:
2024-07-07T18:38:25.833Z info Kustomization/flux-system.flux-system - Source artifact not found, retrying in 30s
2024-07-07T18:38:55.841Z info Kustomization/flux-system.flux-system - Source artifact not found, retrying in 30s
2024-07-07T18:39:19.680Z info Kustomization/flux-system.flux-system - server-side apply for cluster definitions completed
2024-07-07T18:39:19.922Z info Kustomization/flux-system.flux-system - server-side apply completed
2024-07-07T18:39:19.936Z info Kustomization/flux-system.flux-system - Reconciliation finished in 901.757493ms, next run in 10m0s
2024-07-07T18:39:26.199Z info Kustomization/flux-system.flux-system - server-side apply for cluster definitions completed
2024-07-07T18:39:26.324Z info Kustomization/flux-system.flux-system - server-side apply completed
2024-07-07T18:39:26.343Z info Kustomization/flux-system.flux-system - Reconciliation finished in 500.843214ms, next run in 10m0s
2024-07-07T18:39:19.027Z info GitRepository/flux-system.flux-system - stored artifact for commit 'Add Flux sync manifests'

Flux Repository
#

Clone Repository
#

Clone the GitHub repository “flux-repo” created by Flux to a local host:

# Clone the Flux repository
git clone git@gitlab.com:jueklug/flux-repo.git

The folder structure should look like this:

flux-repo
└── clusters
    └── my-cluster
        └── flux-system
            ├── gotk-components.yaml
            ├── gotk-sync.yaml
            └── kustomization.yaml

Install Flux CLI
#

Install the Flux CLI on the host where the Flux repository was cloned:

# Install Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash

Public Deployment Project
#

Add Repository Source
#

Define the GitHub example project “k8s-example-deployment” as source:

# Create repository source
flux create source git k8s-example-deployment \
  --url=https://gitlab.com/jueklug/k8s-example-deployment \
  --branch=main \
  --interval=1m \
  --export > clusters/my-cluster/k8s-example-deployment-source.yaml

Add Repository Kustomization
#

This will configure Flux to build and apply the manifests in the “deployments” directory:

# Create a kustomization
flux create kustomization k8s-example-deployment \
  --target-namespace=default \
  --source=k8s-example-deployment \
  --path="./deployments" \
  --prune=true \
  --wait=true \
  --interval=30m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/my-cluster/k8s-example-deployment-kustomization.yaml

Verify Folder Structure
#

The folder structure should now look like this

# Flux folder structure
clusters
└── my-cluster
    ├── flux-system
    │   ├── gotk-components.yaml
    │   ├── gotk-sync.yaml
    │   └── kustomization.yaml
    ├── k8s-example-deployment-kustomization.yaml
    └── k8s-example-deployment-source.yaml

Push
#

Push the kustomization and source manifests into the Flux GitHub repository:

# configure git
git config --global user.email "juergen@jklug.work"
git config --global user.name "Juergen"

Push repository source:

# Add source manifest and commit
git add clusters/my-cluster/k8s-example-deployment-source.yaml && git commit -m "Add Source"

# Push source to GitLab
git push

Verify Flux source:

# List Flux sources
flux get sources git

# Shell output:
NAME                    REVISION                SUSPENDED       READY   MESSAGE
flux-system             main@sha1:d1b12cc9      False           True    stored artifact for revision 'main@sha1:d1b12cc9'
k8s-example-deployment  main@sha1:dfae5570      False           True    stored artifact for revision 'main@sha1:dfae5570'

Push repository kustomization:

# Add kustomization manifest and commit
git add clusters/my-cluster/k8s-example-deployment-kustomization.yaml && git commit -m "Add Kustomization"

# Push kustomization to GitLab
git push

Verify Flux kustomization:

# List Flux kustomizations
flux get kustomizations

# Shell output: Wait till ready
NAME                    REVISION                SUSPENDED       READY   MESSAGE
flux-system             main@sha1:70b3dcb3      False           True    Applied revision: main@sha1:70b3dcb3
k8s-example-deployment                          False           Unknown Reconciliation in progress

# Shell output:
NAME                    REVISION                SUSPENDED       READY   MESSAGE
flux-system             main@sha1:70b3dcb3      False           True    Applied revision: main@sha1:70b3dcb3
k8s-example-deployment  main@sha1:dfae5570      False           True    Applied revision: main@sha1:dfae5570

Watch Flux Deployment
# 

# Watch the Flux deployment
flux get kustomizations --watch

Verify the Deployment
# 

# List pods in the default namespace
kubectl get pods

# Shell output:
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-7c79c4bf97-rnhgk   1/1     Running   0          2m3s
nginx-deployment-7c79c4bf97-trffr   1/1     Running   0          2m3s
nginx-deployment-7c79c4bf97-xg7w5   1/1     Running   0          2m3s

Stop the Deployment
# 

# Delete kustomization
flux delete kustomization k8s-example-deployment

# Delete source
flux delete source git k8s-example-deployment

# Verify the deletion: List all Sources & Kustomizations
flux get all

# Shell output:
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:70b3dcb3      False           True    stored artifact for revision 'main@sha1:70b3dcb3'

NAME                            REVISION                SUSPENDED       READY   MESSAGE
kustomization/flux-system       main@sha1:70b3dcb3      False           True    Applied revision: main@sha1:70b3dcb3

# Remove the source and kustomization from the repository
rm k8s-example-deployment-kustomization.yaml k8s-example-deployment-source.yaml

# Remove the manifests and commit
git add -A && git commit -m "Remove Source and Kustomization"

# Push changes to GitHub
git push

Private Deployment Project
#

Create SSH Key for Flux
#

On the Kubernetes Controller node create a SSH key pair for Flux:

# Create a Deploy Key for GitHub
ssh-keygen -t rsa -b 4096 -f flux-deploy-key

# Copy the public SSH key
cat flux-deploy-key.pub

Add SSH Key to GitLab Repository
#

Add the SSH public key to the GitLab repository “k8s-example-deployment-private”.

  • Open the GitLab.com repository

  • Go to (Project) “Settings” > “Repository”

  • Scroll down to “Deploy keys” & click “Expand”

  • Click “Add new key”

  • Add a title for the key like Flux-Key

  • Paste the public SSH key

# Public SSH key
ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCG3YyUgG1kseSq875GWW1lDkyULSrkjUF5JAqu...
  • Click “Add key”

Kubernetes Secret
#

Create a Kubernetes secret with the Flux SSH private key. The following command also adds the host key for the GitLab repository:

# Create Kubernetes secret (Run command in same dir as the SSH key)
flux create secret git flux-git-deploy \
  --url=ssh://git@gitlab.com/jueklug/k8s-example-deployment-private \
  --private-key-file=./flux-deploy-key \
  --namespace=flux-system

# Delete Kubernetes secret
kubectl delete secret flux-git-deploy -n flux-system

# Optional: Verify / decode the Kubernetes secret
kubectl get secret flux-git-deploy -n flux-system -o jsonpath="{.data.identity}" | base64 --decode

Add Repository Source
#

Define the GitLab example project “k8s-example-deployment” as source:

# Create repository source
flux create source git k8s-example-deployment-private \
  --url=ssh://git@gitlab.com/jueklug/k8s-example-deployment-private \
  --branch=main \
  --interval=1m \
  --secret-ref=flux-git-deploy \
  --export > clusters/my-cluster/k8s-example-deployment-private-source.yaml

Add Repository Kustomization
#

This will configure Flux to build and apply the manifests in the “deployments” directory:

# Create a kustomization
flux create kustomization k8s-example-deployment-private \
  --target-namespace=default \
  --source=k8s-example-deployment-private \
  --path="./deployments" \
  --prune=true \
  --wait=true \
  --interval=30m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/my-cluster/k8s-example-deployment-private-kustomization.yaml

Verify Folder Structure
#

The folder structure should now look like this

# Flux folder structure
clusters
└── my-cluster
    ├── flux-system
    │   ├── gotk-components.yaml
    │   ├── gotk-sync.yaml
    │   └── kustomization.yaml
    ├── k8s-example-deployment-private-kustomization.yaml
    └── k8s-example-deployment-private-source.yaml

Push
#

Push the kustomization and source manifests into the Flux GitLab repository:

# configure git
git config --global user.email "juergen@jklug.work"
git config --global user.name "Juergen"

Push repository source:

# Add source manifest and commit
git add clusters/my-cluster/k8s-example-deployment-private-source.yaml && git commit -m "Add Source"

# Push source to GitLab
git push

Verify Flux source:

# List Flux sources
flux get sources git

# Shell output:
NAME                            REVISION                SUSPENDED       READY   MESSAGE
flux-system                     main@sha1:63365ae5      False           True    stored artifact for revision 'main@sha1:63365ae5'
k8s-example-deployment-private  main@sha1:dd794655      False           True    stored artifact for revision 'main@sha1:dd794655'

Push repository kustomization:

# Add kustomization manifest and commit
git add clusters/my-cluster/k8s-example-deployment-private-kustomization.yaml && git commit -m "Add Kustomization"

# Push kustomization to GitLab
git push

Verify Flux kustomization:

# List Flux kustomizations
flux get kustomizations

# Shell output: Wait till ready
NAME                            REVISION                SUSPENDED       READY   MESSAGE
flux-system                     main@sha1:0917ccf4      False           True    Applied revision: main@sha1:0917ccf4
k8s-example-deployment-private  main@sha1:dd794655      False           True    Applied revision: main@sha1:dd794655

Verify the Deployment
# 

# List pods in the default namespace
kubectl get pods

# Shell output:
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-deployment-private-7c79c4bf97-5qxbf   1/1     Running   0          83s
nginx-deployment-private-7c79c4bf97-7j4dt   1/1     Running   0          83s
nginx-deployment-private-7c79c4bf97-jrwhr   1/1     Running   0          83s

Stop the Deployment
# 

# Delete kustomization
flux delete kustomization k8s-example-deployment-private

# Delete source
flux delete source git k8s-example-deployment-private

# Verify the deletion: List all Sources & Kustomizations
flux get all

# Shell output:
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:0917ccf4      False           True    stored artifact for revision 'main@sha1:0917ccf4'

NAME                            REVISION                SUSPENDED       READY   MESSAGE
kustomization/flux-system       main@sha1:0917ccf4      False           True    Applied revision: main@sha1:0917ccf4

# Remove the source and kustomization from the repository
rm k8s-example-deployment-private-kustomization.yaml k8s-example-deployment-private-source.yaml

# Remove the manifests and commit
git add -A && git commit -m "Remove Source and Kustomization"

# Push changes to GitLab
git push
Flux - This article is part of a series.
Part 1: Deploying with Flux: Deploy Manifests from public and private GitHub Repositories into Kubernetes Cluster, Flux Commands
Part 2: This Article
Part 3: Deploying with Flux: Deploy Manifests from private Repositories of a self-hosted GitLab Instance into Kubernetes Cluster