↓Skip to main content

Deploying with Argo CD: Push Custom Container to GitLab Registry & Deploy the Container with Argo CD into K8s Cluster

1297 words·
Argo CD GitLab GitLab CI Kubernetes K8s
Table of Contents
ArgoCD - This article is part of a series.
Part 1: Deploying with Argo CD: Installation, CLI Setup, Helm Chart Deployment from GitLab Repository in Kubernetes Cluster
Part 2: This Article

Overview
#

I’m using a Kubeadm based K8s cluster with one controller and two worker nodes, with MetalLB and Nginx Ingress.

# Kubernetes Cluster
NAME      STATUS   ROLES           AGE     VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu1   Ready    control-plane   9m38s   v1.28.11   192.168.30.10   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18
ubuntu2   Ready    worker          7m19s   v1.28.11   192.168.30.11   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18
ubuntu3   Ready    worker          7m15s   v1.28.11   192.168.30.12   <none>        Ubuntu 24.04 LTS   6.8.0-35-generic   containerd://1.7.18

192.168.70.4 # Local GitLab Instance "gitlab.jklug.work"
192.168.70.5 # GitLab Runner

Argo CD CLI
#

I’m installing the Argo CD CLI in my Kubernetes Controller node.

Installation
#

Find the latest stable release:
https://github.com/argoproj/argo-cd/tags

# Install Argo CD CLI
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/download/v2.11.4/argocd-linux-amd64

# Change permissions
chmod +x argocd

# Move the binary
sudo mv argocd /usr/local/bin/

# Verify the installation / check version
argocd version

Configuration
#

Argo CD DNS Entry
#

Make sure the host where the Argo CD CLI is deploy can resolve the DNS name of the Argo CD server:

# Add DNS entry to /etc/hosts
sudo tee -a /etc/hosts <<EOF
192.168.30.201 argocd.jklug.work
EOF

Set Argo CD Server Address
# 

# Set the the Argo CD server address
argocd login argocd.jklug.work

# Shell output:
Username: admin
Password: # Enter the admin pw
'admin:login' logged in successfully
Context 'argocd.jklug.work' updated

Kubernetes Prerequisites
#

GitLab DNS Entry
#

Add DNS entries for GitLab & the GitLab registry to the Kubernetes cluster nodes:

# Add DNS entry to /etc/hosts
sudo tee -a /etc/hosts <<EOF
192.168.70.4 gitlab.jklug.work
192.168.70.4 gitlab-registry.jklug.work
EOF

CoreDNS ConfigMap
#

Backup the ConfigMap
# 

# Export the current ConfigMap
kubectl get cm coredns -n kube-system -o yaml > coredns-configmap-backup.yaml

Add GitLab DNS Entry
#

Create a DNS entry that points to GitLab:

# Add the following DNS section to the CoreDNS ConfigMap
hosts {
    192.168.70.4 gitlab.jklug.work
    192.168.70.4 gitlab-registry.jklug.work
    fallthrough
}

# Edit the CoreDNS ConfigMap
kubectl edit cm coredns -n kube-system

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        hosts {
            192.168.70.4 gitlab.jklug.work
            192.168.70.4 gitlab-registry.jklug.work
            fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }    
kind: ConfigMap
metadata:
  creationTimestamp: "2024-07-05T17:36:16Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "224"
  uid: 0c9d9c5a-c49f-4392-a465-ffb1d047811c

# Restart CoreDNS
kubectl rollout restart deployment coredns -n kube-system

Verify DNS Resolution
# 

# Run a busybox pod
kubectl run busybox --image=busybox --restart=Never --stdin --tty

# Test the GitLab DNS resolution
nslookup gitlab.jklug.work

# Shell output:
Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   gitlab.jklug.work
Address: 192.168.70.4

# Exit the container terminal
exit

# Delete the busybox pod
kubectl delete pod busybox

Argo CD Prerequisites
#

Scan GitLab Host Keys
#

From the node where the Argo CLI is installed (or any other host that can resolve the GitLab DNS name), scan the host keys of GitLab:

# List the GitLab host keys
ssh-keyscan gitlab.jklug.work

Shell output:

# gitlab.jklug.work:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
# gitlab.jklug.work:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
gitlab.jklug.work ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHAQj15hmfv3OwTY3RAPwx1UlZ8p4qgtAHiZ9hngfJTSScO1kf40oL3Ek5NVKSGYZQ4w6ozBFHKO3l6tHn8nPeD7mUk/nW2U5w9yYpcRyFknn/u/Z0QreHAkI8fg6LI4n+2QYFF1rZbIemtCG33FozwrWKJ+/UsJYLnuQ2fenjcvkwPYx7NKV07RtQ3xYvkFVdWQGFJK8pLG9UcsanwZbH2nVPbv3i9KKI9xxWmJDh9JOoLhG6JipNN4Q4CoodfR9k9A2PY88dEykMInSGzFddOqbHLyISO8H1oJrofPzovPR07f+bDBK6iGIqRSW00k6mM0RFkPPo9tulLJ87DgB84jVrtYGp71wmV9PQ8jPB1uaDx5JtRNc0G+IWlIzTy8hFW9djELdTdQmfxeaCceyn1AmuXhpwZin64WTqztXj29s1olZ0+Uchh2FGpEjvlVqveeMmgAQkQezidqVHKKwinW1zdeSaBaZkS0JpLtxNpA86vBnhtYE8Z4CaQAvoQXU=
# gitlab.jklug.work:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
gitlab.jklug.work ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUxIo1glUPlmYJDbAOvHlRd/qjxdIEJCtBlcFLCMXECbRdp9IN/qePZdFtOnMWWVNvi8qy+7V8XbIFbzHoYwcg=
# gitlab.jklug.work:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
gitlab.jklug.work ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH6GrjO8VqbiFMwtOfaEuKd3bV2vb7jH4r5Xl9PW1TFY
# gitlab.jklug.work:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7

Add GitLab Host Keys to Argo CD
#

Open the Argo CD webinterface an add the GitLab host keys:

  • Go to: “Settings” > “Repository certificates and known hosts” > “ADD SSH KNOWN HOSTS”
  • Paste the GitLab SSH keys in the text box, one entry per line
  • Click “CREATE” to save the entry

GitLab
#

Argo CLI SSH Key
#

Create a SSH key on the Argo CD CLI host and add the public key to GitLab:

# Create RSA Key: 4096 bit
ssh-keygen -t rsa -b 4096

Add SSH Key to GitLab
#

  • Go to: (Profile) > “Edit profile” > “SSH Keys”

  • Click “Add new key”

  • Paste the public SSH key

  • Define a title like “Argo CLI”

  • Click “Add key”


GitLab Deployment Repository
#

Folder Structure
#

The folder structure of the GitLab repository looks like this:

├── deployments
│   └── custom-nginx-deployment.yaml
├── Dockerfile
└── website
    └── index.html

Dockerfile
# 

Dockerfile

FROM nginx:1.18
COPY website/. /usr/share/nginx/html
EXPOSE 80

HTML File
# 

website/index.html

<!DOCTYPE html>
<html>

<head>
	<title>Gitlab Container</title>
</head>

<body>
	<h1>Nginx Container for K8s</h1>
	<p>Kubernetes Deployment</p>
</body>

</html>

CI Pipeline
# 

.gitlab-ci.yml

stages:
  - build

build:
  stage: build
  image: docker:20.10.16
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  services:
    - docker:20.10.16-dind

  script:
    - docker build -t $CI_REGISTRY_IMAGE:latest .
    - docker tag $CI_REGISTRY_IMAGE:latest $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

Deployment Manifest
# 

deployments/custom-nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-nginx-deployment
  namespace: default
spec:
  selector:
    matchLabels:
      app: custom-nginx
  replicas: 5
  template:
    metadata:
      labels:
        app: custom-nginx
    spec:
      containers:
      - name: custom-nginx
        image: gitlab-registry.jklug.work/root/argocd-gitlabci:main
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: gitlab-registry-secret

---
apiVersion: v1
kind: Service
metadata:
  name: custom-nginx-deployment
  namespace: default
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: custom-nginx

CI Pipeline Logs
#

Check the GitLab CI pipeline logs:

...
$ docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
The push refers to repository [gitlab-registry.jklug.work/root/argocd-gitlabci]
27a50e29d6d4: Preparing
4fa6704c8474: Preparing
4fe7d87c8e14: Preparing
6fcbf7acaafd: Preparing
f3fdf88f1cb7: Preparing
7e718b9c0c8c: Preparing
7e718b9c0c8c: Waiting
27a50e29d6d4: Pushed
4fa6704c8474: Pushed
6fcbf7acaafd: Pushed
4fe7d87c8e14: Pushed
f3fdf88f1cb7: Pushed
7e718b9c0c8c: Pushed
main: digest: sha256:e5bfc112504cfb3ec64e6c5cb09f9c261d15145b4008d128baad3411477f79e5 size: 1569
Cleaning up project directory and file based variables
00:01
Job succeeded

GitLab Registry Token
#

Create a GitLab Registry token, used for Argo CD to access the GitLab Registry:

  • Go to: Profile > Access Tokens

  • Click “Add new token”

  • Define a token name like ArgoCD

  • Define the scope of the token: “read_registry” Grants read-only access to container registry images on private projects.

  • Click “Create personal access token”

# Copy the token
glpat-x8LDGfwZYJrmp5yDmqCr

Test GitLab Registry Connection
#

Optional: Test the GitLab Registry connection. This can be done from any host that can resolve the GitLab Registry DNS name and has Docker installed.

# Manually connect to the registry: Syntax
docker login gitlab-registry.jklug.work -u <your-gitlab-username> -p <your-personal-access-token>

# Manually connect to the registry: Example
docker login gitlab-registry.jklug.work -u root -p glpat-x8LDGfwZYJrmp5yDmqCr

# Shell output:
Login Succeeded

Optional: Manually pull the image from the GitLab registry.

# Manually pull the image
docker pull gitlab-registry.jklug.work/root/argocd-gitlabci:main

Deploy with Argo CD
#

GitLab Registry Secret
# 

# Create a secret with the GitLab access token
kubectl create secret docker-registry gitlab-registry-secret \
--docker-server=gitlab-registry.jklug.work \
--docker-username=root \
--docker-password=glpat-x8LDGfwZYJrmp5yDmqCr \
--docker-email=juergen@jklug.work \
--namespace=default

# Shell output:
secret/gitlab-registry-secret created

Verify the Secret
# 

# List secret details
kubectl get secret gitlab-registry-secret -o yaml

Connect GitLab Repository:
# 

# Connect a GitLab repository via SSH
argocd repo add git@gitlab.jklug.work:root/argocd-gitlabci.git --ssh-private-key-path /home/ubuntu/.ssh/id_rsa

# Shell output:
Repository 'git@gitlab.jklug.work:root/argocd-gitlabci.git' added

The GitLab repository should now be availble in the Argo CD webinterface under: “Settings” > “Repositories”


Create an Application in Argo CD
#

Create an Argo CD application named “argocd-gitlabci” that points to the Kubernetes manifests in the specified GitLab repository and deploys them to the default namespace of the Kubernetes cluster.

# Deploy the GitLab repository
argocd app create argocd-gitlabci \
  --repo git@gitlab.jklug.work:root/argocd-gitlabci.git \
  --path deployments \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default

# Shell output:
application 'argocd-gitlabci' created

The application should now be available in the “Applications” section of the Argo CD webinterface.


Sync / Deploy the Application
# 

# Deploy resources into the Kubernetes cluster
argocd app sync argocd-gitlabci

# Undeploy
argocd app delete argocd-gitlabci

Verify the Deployment
# 

# List pods in "default" namespace
kubectl get pods

# Shell output
NAME                                       READY   STATUS    RESTARTS   AGE
custom-nginx-deployment-7545cf6cd4-686ml   1/1     Running   0          8s
custom-nginx-deployment-7545cf6cd4-8kwfp   1/1     Running   0          8s
custom-nginx-deployment-7545cf6cd4-dsc8k   1/1     Running   0          8s
custom-nginx-deployment-7545cf6cd4-px9rz   1/1     Running   0          8s
custom-nginx-deployment-7545cf6cd4-qdz5c   1/1     Running   0          8s

# List services in "default" namespace
kubectl get svc

# Shell output
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
custom-nginx-deployment   LoadBalancer   10.100.255.19   192.168.30.202   80:32477/TCP   7s
kubernetes                ClusterIP      10.96.0.1       <none>           443/TCP        3d2h

# Curl the LoadBalancer service
curl 192.168.30.202:80

# Shell output:
<!DOCTYPE html>
<html>

<head>
        <title>Gitlab Container</title>
</head>

<body>
        <h1>Nginx Container for K8s</h1>
        <p>Kubernetes Deployment</p>
</body>

</html>
ArgoCD - This article is part of a series.
Part 1: Deploying with Argo CD: Installation, CLI Setup, Helm Chart Deployment from GitLab Repository in Kubernetes Cluster
Part 2: This Article