↓Skip to main content

Argo CD Dynamic Environments with GitLab, Harbor and Ansible

1625 words·
Argo CD GitLab Harbor Ansible Dynamic Environments Pull Request Generator GitOps Kubernetes
Table of Contents

Overview
#

The Argo CD pull request generator uses the GitLab API to discover open merge requests in a GitLab repository. This enables temporary deployments of feature branches for preview and testing purposes.


Prerequisites
#

SSH Key Pair
#

Create a private SSH key pair, that is used by Argo CD to access the GitLab Helm Chart repository:

# Create SSH key pair
ssh-keygen -t rsa -b 2048 -C "deploy_key" -f ~/.ssh/deploy_key

  • deploy_key Private SSH key, used by Argo CD

  • deploy_key.pub Public SSH key, add to GitLab Helm repository as deploy key


Harbor Project and Credentials
#

I’m using the following Harbor project and credentials:

Harbor Project name: dynamic-environmnets-1

User: robot$dynamic-environmnets-1+dynamic-env-admin

Password: RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce



GitLab Code Repository
#

File and Folder Structure
# 

# dyn-env-code
├── Dockerfiles
│   └── Dockerfile
├── .gitlab-ci.yml
├── README.md
└── website
    └── index.html

CI/CD Variables
#

Add the credentials for the Harbor Project as GitLab CI/CD Variables:

# Key
HARBOR_USERNAME 

# Value
robot$dynamic-environmnets-1+dynamic-env-admin

# Key
HARBOR_PASSWORD

# Value
RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce

Variable options:

  • Enable “Visibility” x “Masked”

  • Disable Flag “Protect variable”


Project Access Token
#

Create a Project access token for Argo CD:

  • Go to: “Settings” > “Access tokens”

  • Click “Add new token”

  • Token name “argocd”

  • Select role “Developer”

  • Scope “read_api”

  • “Click Create project access token”

  • Copy the token: glpat-49lc27dJBfr6zYKYH9egQm86MQp1OjYH.01.0w0z21y50


Pipeline Manifest
#

  • .gitlab-ci.yml
variables:
  HARBOR_CONTAINER_REGISTRY: "harbor.jklug.work"

stages:
  - build

build-preview-image:
  stage: build
  image: docker:28.4
  services:
    - name: docker:28.4-dind
      alias: docker
      command: ["--tls=false"]
  variables:
    DOCKER_TLS_CERTDIR: ""
    DEPLOYMENT_ENVIRONMENT: preview
    VERSION: "${CI_MERGE_REQUEST_IID}_${CI_COMMIT_SHORT_SHA}"
  before_script:
    # Login to Harbor registry via robot user
    - docker login -u "$HARBOR_USERNAME" -p "$HARBOR_PASSWORD" "$HARBOR_CONTAINER_REGISTRY"
  script:
    # Define image tag name
    - image="${HARBOR_CONTAINER_REGISTRY}/dynamic-environmnets-1/${DEPLOYMENT_ENVIRONMENT}/website-preview:${VERSION}"
    # Build and tag image
    - docker build -f Dockerfiles/Dockerfile --pull -t "${image}" .
    # Push image to Harbor
    - docker push "${image}"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Dockerfile
#

  • Dockerfiles/Dockerfile
FROM --platform=linux/amd64 nginx:1.18

COPY website/index.html /usr/share/nginx/html

Example Website
#

  • website/index.html
<!DOCTYPE html>
<html>

<head>
        <title>K8s Argo CD</title>
</head>

<body>
  <h1>Argo CD Dynamic Environments</h1>
  <p>Example Deployment - feature 2</p>
</body>

</html>



GitLab Helm Repository
#

File and Folder Structure
# 

# dyn-env-helm
├── helm-chart
│   ├── Chart.yaml
│   ├── templates
│   │   ├── deployment.yaml
│   │   ├── harbor-registry-secret.yaml
│   │   ├── ingress.yaml
│   │   ├── namespace.yaml
│   │   └── service.yaml
│   └── values.yaml
└── README.md

Add Deploy Key
#

Add the previously created public SSH key, so that Argo CD can access the Helm chart repository:

  • Go to: (Settings) > “Repository” > “Deploy keys”

  • Click “Add new key”

  • Define a title like “argocd_deploy”

  • Paste the public SSH key into the “Key” section

  • Disable “Grant write permissions to this key”

  • Click “Add key”


Helm Chart
#

values.yaml
#

  • helm-chart/values.yaml
example_app:
  repository: ""
  tag: main_0
  ingress_domain: dynamic-environments.jklug.work
  container_target_port: 80
imagePullSecrets:
  - harbor-registry-secret # The Kubernetes secret name
harbor:
  server: harbor.jklug.work
  username: ""
  password: ""

Chart.yaml
#

  • helm-chart/Chart.yaml
apiVersion: v2
name: Dynamic Environments
description: Helm chart for Dynamic Environments Playground
type: application
version: "0"
appVersion: "0"
message: |
    first commit

namespace.yaml
#

  • templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

deployment.yaml
#

  • templates/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamic-environments-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamic-environments
  template:
    metadata:
      labels:
        app: dynamic-environments
    spec:
      imagePullSecrets:
        - name: "{{ .Values.imagePullSecrets | first }}"
      containers:
        - name: dynamic-environments
          image: "{{ .Values.example_app.repository }}:{{ .Values.example_app.tag }}"
          ports:
            - containerPort: {{ .Values.example_app.container_target_port | int }}
          imagePullPolicy: IfNotPresent

service.yaml
#

  • templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: dynamic-environments-service
spec:
  type: ClusterIP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 900
  selector:
    app: dynamic-environments
  ports:
    - name: http
      protocol: TCP
      port: {{ .Values.example_app.container_target_port | int }}
      targetPort: {{ .Values.example_app.container_target_port | int }}

ingress.yaml
#

  • templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dynamic-environments-ingress
  annotations:
    #cert-manager.io/cluster-issuer: cluster-issuer
    nginx.ingress.kubernetes.io/ssl-redirect: "false"  # Set to true in prod
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx-metallb
  #tls:
    #- hosts:
        #- "{{ .Values.example_app.ingress_domain }}"
      #secretName: "example-app-tls-secret"
  rules:
    - host: "{{ .Values.example_app.ingress_domain }}"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: dynamic-environments-service
                port:
                  number: {{ .Values.example_app.container_target_port | int }}

harbor-registry-secret.yaml
#

Add the Harbor project credentials to the Kubernetes secret:

# Create the Harbor authentication string
REGISTRY='harbor.jklug.work'
USER='robot$dynamic-environmnets-1+dynamic-env-admin'
PASS='RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce'

AUTH="$(printf '%s:%s' "$USER" "$PASS" | base64 -w0)"

JSON="$(cat <<EOF
{"auths":{"$REGISTRY":{"username":"$USER","password":"$PASS","auth":"$AUTH"}}}
EOF
)"

printf '%s' "$JSON" | base64 -w0
echo

# Shell output:
eyJhdXRocyI6eyJoYXJib3IuamtsdWcud29yayI6eyJ1c2VybmFtZSI6InJvYm90JGR5bmFtaWMtZW52aXJvbm1uZXRzLTErZHluYW1pYy1lbnYtYWRtaW4iLCJwYXNzd29yZCI6IlJYT2FheDZGTDdXdFg5V0Y3ZVhLUnlkUWxFMFIyVWNlIiwiYXV0aCI6ImNtOWliM1FrWkhsdVlXMXBZeTFsYm5acGNtOXViVzVsZEhNdE1TdGtlVzVoYldsakxXVnVkaTFoWkcxcGJqcFNXRTloWVhnMlJrdzNWM1JZT1ZkR04yVllTMUo1WkZGc1JUQlNNbFZqWlE9PSJ9fX0=

# Verify / decode
echo 'eyJhdXRocyI6eyJoYXJib3IuamtsdWcud29yayI6eyJ1c2VybmFtZSI6InJvYm90JGR5bmFtaWMtZW52aXJvbm1uZXRzLTErZHluYW1pYy1lbnYtYWRtaW4iLCJwYXNzd29yZCI6IlJYT2FheDZGTDdXdFg5V0Y3ZVhLUnlkUWxFMFIyVWNlIiwiYXV0aCI6ImNtOWliM1FrWkhsdVlXMXBZeTFsYm5acGNtOXViVzVsZEhNdE1TdGtlVzVoYldsakxXVnVkaTFoWkcxcGJqcFNXRTloWVhnMlJrdzNWM1JZT1ZkR04yVllTMUo1WkZGc1JUQlNNbFZqWlE9PSJ9fX0=' | base64 -d

# Shell output:
{"auths":{"harbor.example.com":{"username":"robot$dynamic-environmnets-1+dynamic-env-admin","password":"RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce","auth":"cm9ib3QkZHluYW1pYy1lbnZpcm9ubW5ldHMtMStkeW5hbWljLWVudi1hZG1pbjpSWE9hYXg2Rkw3V3RYOVdGN2VYS1J5ZFFsRTBSMlVjZQ=="}}}
  • harbor-registry-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: harbor-registry-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJoYXJib3IuamtsdWcud29yayI6eyJ1c2VybmFtZSI6InJvYm90JGR5bmFtaWMtZW52aXJvbm1uZXRzLTErZHluYW1pYy1lbnYtYWRtaW4iLCJwYXNzd29yZCI6IlJYT2FheDZGTDdXdFg5V0Y3ZVhLUnlkUWxFMFIyVWNlIiwiYXV0aCI6ImNtOWliM1FrWkhsdVlXMXBZeTFsYm5acGNtOXViVzVsZEhNdE1TdGtlVzVoYldsakxXVnVkaTFoWkcxcGJqcFNXRTloWVhnMlJrdzNWM1JZT1ZkR04yVllTMUo1WkZGc1JUQlNNbFZqWlE9PSJ9fX0=



Ansible Playbook
#

Apply ApplicationSet
# 

---
# Deploy Dynamic Environments ApplicationSet - Playground
- name: Deploy Dynamic Environments ApplicationSet - Playground
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    # Vault Secrets
    argocd_repository_keys: "{{ lookup('community.hashi_vault.vault_kv2_get', 'services/argocd/gitlab_deploykey', engine_mount_point='homelab_prod').secret }}"
    # Vault Variables
    argocd_repository_private_key: "{{ argocd_repository_keys.private_key }}"  # Key to access GitLab Helm chart repository
    # GitLab Helm Chart Repository
    gitlab_helm_repository: "git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git"  # Define GitLab Helm chart repository
    gitlab_repository_name: "dynamic-environments-helm"  # Just a unique name for the GitLab repository in Argo CD
    # GitLab Code Repository
    project_access_token: "glpat-49lc27dJBfr6zYKYH9egQm86MQp1OjYH.01.0w0z21y50"
    # Harbor Credentials
    harbor_user: "robot$dynamic-environmnets-1+dynamic-env-admin"
    harbor_pw: "XOaax6FL7WtX9WF7eXKRydQlE0R2Uce"
    # Argo CD Variables
    argocd_namespace: "argocd"  # Argo CD default namespace

  tasks:
    - name: Add GitLab Helm Repository via Argo CD Repository Secret
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: "{{ gitlab_repository_name }}"
            namespace: "{{ argocd_namespace }}"
            labels:
              argocd.argoproj.io/secret-type: repository
          type: Opaque
          stringData:
            url: "{{ gitlab_helm_repository }}"
            sshPrivateKey: "{{ argocd_repository_private_key }}"

    - name: Add GitLab Code Repository Secret
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: dynamic-environments-api
            namespace: "{{ argocd_namespace }}"
          type: Opaque
          stringData:
            token: "{{ project_access_token }}"  # scope read_api, role developer

    - name: Create Argo CD AppProject
      kubernetes.core.k8s:
        state: present
        namespace: argocd
        definition: |
          apiVersion: argoproj.io/v1alpha1
          kind: AppProject
          metadata:
            name: dynamic-environments-playground
            namespace: argocd
          spec:
            description: Playground for Dynamic Preview Environments
            sourceRepos:
              - '{{ gitlab_helm_repository }}'
            destinations:
              - namespace: '*'  # Allow deployment to any namespace
                server: '*'  # Allow deployment to any cluster
            clusterResourceWhitelist:
              - group: ''  # Core API group
                kind: Namespace          

    - name: Apply Argo CD ApplicationSet
      kubernetes.core.k8s:
        state: present
        namespace: argocd
        definition: |
          apiVersion: argoproj.io/v1alpha1
          kind: ApplicationSet
          metadata:
            name: dynamic-environments
            namespace: argocd
          spec:
            goTemplate: true  # Enables variable references
            generators:
              - pullRequest:
                  requeueAfterSeconds: 30  # Poll for changes
                  gitlab:
                    api: https://gitlab.jklug.work
                    project: "3"  # Code repository project ID
                    tokenRef:
                      secretName: dynamic-environments-api  # Kubernetes secret name
                      key: token
                    labels: ["preview"]
                    pullRequestState: opened
            template:
              metadata:
                name: "{% raw %}review-app-{{ .number }}{% endraw %}"
              spec:
                project: default
                source:
                  repoURL: '{{ gitlab_helm_repository }}'  # Helm repository URL
                  targetRevision: HEAD  # Or a stable branch/tag
                  path: helm-chart  # Path to Helm chart
                  helm:
                    parameters:
                      - name: example_app.repository
                        value: 'harbor.jklug.work/dynamic-environmnets-1/preview/website-preview'
                      - name: example_app.tag
                        value: "{% raw %}{{ .number }}_{{ .head_short_sha }}{% endraw %}"
                      - name: example_app.ingress_domain
                        value: "{% raw %}{{ .branch_slug }}-{{ .number }}.dynamic-environments.jklug.work{% endraw %}"
                      - name: harbor.username
                        value: '{{ harbor_user }}'
                        forceString: true
                      - name: harbor.password
                        value: '{{ harbor_pw }}'
                        forceString: true
                destination:
                  server: https://kubernetes.default.svc
                  namespace: "{% raw %}argocdplayground-{{ .branch_slug }}-{{ .number }}{% endraw %}"
                syncPolicy:
                  automated:
                    prune: true
                    selfHeal: true
                  syncOptions:
                    - PruneLast=true  # Prune on deletion          

# Run Ansible playbook: 
ansible-playbook playbooks/argocd_dynamic_environments.yml -i inventory



GitLab Code Repository
#

Triger Dynamic Deployment
# 

# Create a newfeature branch
git checkout -b feature/feature102

# Add a change
vi website/index.html
git add website/index.html

# Commit the change
git commit -m "feature102"

# Push the change
git push -u origin feature/feature102

Useful Git Commands:

# Delete the local feature branch
git branch -d feature/feature101

# Delete the remote feature branch
git push origin --delete feature/feature101

Create Merge Request
#

Create a merge request with the label “preview”:

Add the “preview” label:

Verify the GitLab pipeline ran throigh:



Argo CD
#

Verify Application
#

After the Merge request was created, Argo CD creates an application:

After the merge request was merged or closed, Argo CD automatically removes the application.



Arg oCD CLI: Verify Resources
#

CLI Login
# 

# Login to Argo CD instance
argocd login argocd.jklug.work --grpc-web

# Default user
admin

List GitLab Repositories
#

The GitLab Helm chart repository is connected via the dynamic-environments-helm Kubernetes secret, to remove the GitLab repository, delete the Kubernetes secret.

# List GitLab repositories
argocd repo list

# Shell output:
TYPE  NAME  REPO                                                                           INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git         git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git  false     false  false  false  Successful

List Project
# 

# List Argo CD projects
argocd proj list

# Shell output:
default                                                                       *,*           *                                                                              */*                         <none>                        <none>          disabled            <none>
dynamic-environments-playground  Playground for Dynamic Preview Environments  *,*           git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git  /Namespace                  <none>                        <none>          disabled            <none>

# Delete project
argocd proj delete dynamic-environments-playground

List ApplicationSet
# 

# List Argo CD ApplicationSets
kubectl -n argocd get applicationsets

# Shell output:
NAME                   AGE
dynamic-environments   32s

# Delete Argo CD ApplicationSet
kubectl -n argocd delete applicationset dynamic-environments

List Application
# 

# Verify Argo CD Applications
argocd app list

# Shell output:
NAME                 CLUSTER                         NAMESPACE                              PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                                           PATH        TARGET
argocd/review-app-3  https://kubernetes.default.svc  argocdplayground-feature-feature102-3  default  Synced  Healthy  Auto-Prune  <none>      git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git  helm-chart  HEAD

# List application details
argocd app get argocd/review-app-3

# Shell output:
Name:               argocd/review-app-3
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          argocdplayground-feature-feature102-3
URL:                https://argocd.jklug.work/applications/review-app-3
Source:
- Repo:             git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git
  Target:           HEAD
  Path:             helm-chart
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to HEAD (f992673)
Health Status:      Healthy

GROUP              KIND        NAMESPACE                              NAME                                   STATUS     HEALTH   HOOK  MESSAGE
                   Namespace   argocdplayground-feature-feature102-3  argocdplayground-feature-feature102-3  Succeeded  Synced         namespace/argocdplayground-feature-feature102-3 created
                   Secret      argocdplayground-feature-feature102-3  harbor-registry-secret                 Synced                    secret/harbor-registry-secret created
                   Service     argocdplayground-feature-feature102-3  dynamic-environments-service           Synced     Healthy        service/dynamic-environments-service created
apps               Deployment  argocdplayground-feature-feature102-3  dynamic-environments-deployment        Synced     Healthy        deployment.apps/dynamic-environments-deployment created
networking.k8s.io  Ingress     argocdplayground-feature-feature102-3  dynamic-environments-ingress           Synced     Healthy        ingress.networking.k8s.io/dynamic-environments-ingress created
                   Namespace                                          argocdplayground-feature-feature102-3  Synced



Kubectl
#

Verify Kubernetes Resources
# 

# List default resources
kubectl get all -n argocdplayground-feature-feature102-3

# Shell output:
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/dynamic-environments-deployment-7c5744ccf9-xzb7g   1/1     Running   0          6m55s

NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/dynamic-environments-service   ClusterIP   10.201.224.241   <none>        80/TCP    6m55s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dynamic-environments-deployment   1/1     1            1           6m55s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/dynamic-environments-deployment-7c5744ccf9   1         1         1       6m55s

# List Ingress resource
kubectl get ingress -n argocdplayground-feature-feature102-3

# Shell output:
NAME                           CLASS           HOSTS                                                  ADDRESS          PORTS   AGE
dynamic-environments-ingress   nginx-metallb   feature-feature102-3.dynamic-environments.jklug.work   192.168.70.110   80      7m36s