↓Skip to main content

Argo CD Dynamic Environments with GitLab, Harbor and Ansible

1625 words·
Argo CD GitLab Harbor Ansible Dynamic Environments Pull Request Generator GitOps
Table of Contents

Overview
#

The Argo CD pull request generator uses the GitLab API to discover open merge requests in a GitLab repository. This enables temporary deployments of feature branches for preview and testing purposes.


Prerequisites
#

SSH Key Pair
#

Create a private SSH key pair, that is used by Argo CD to access the GitLab Helm Chart repository:

# Create SSH key pair
ssh-keygen -t rsa -b 2048 -C "deploy_key" -f ~/.ssh/deploy_key

  • deploy_key Private SSH key, used by Argo CD

  • deploy_key.pub Public SSH key, add to GitLab Helm repository as deploy key


Harbor Project and Credentials
#

I’m using the following Harbor project and credentials:

Harbor Project name: dynamic-environmnets-1

User: robot$dynamic-environmnets-1+dynamic-env-admin

Password: RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce



GitLab Code Repository
#

File and Folder Structure
# 

# dyn-env-code
├── Dockerfiles
│   └── Dockerfile
├── .gitlab-ci.yml
├── README.md
└── website
    └── index.html

CI/CD Variables
#

Add the credentials for the Harbor Project as GitLab CI/CD Variables:

# Key
HARBOR_USERNAME 

# Value
robot$dynamic-environmnets-1+dynamic-env-admin

# Key
HARBOR_PASSWORD

# Value
RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce

Variable options:

  • Enable “Visibility” x “Masked”

  • Disable Flag “Protect variable”


Project Access Token
#

Create a Project access token for Argo CD:

  • Go to: “Settings” > “Access tokens”

  • Click “Add new token”

  • Token name “argocd”

  • Select role “Developer”

  • Scope “read_api”

  • “Click Create project access token”

  • Copy the token: glpat-49lc27dJBfr6zYKYH9egQm86MQp1OjYH.01.0w0z21y50


Pipeline Manifest
#

  • .gitlab-ci.yml
variables:
  HARBOR_CONTAINER_REGISTRY: "harbor.jklug.work"

stages:
  - build

build-preview-image:
  stage: build
  image: docker:28.4
  services:
    - name: docker:28.4-dind
      alias: docker
      command: ["--tls=false"]
  variables:
    DOCKER_TLS_CERTDIR: ""
    DEPLOYMENT_ENVIRONMENT: preview
    VERSION: "${CI_MERGE_REQUEST_IID}_${CI_COMMIT_SHORT_SHA}"
  before_script:
    # Login to Harbor registry via robot user
    - docker login -u "$HARBOR_USERNAME" -p "$HARBOR_PASSWORD" "$HARBOR_CONTAINER_REGISTRY"
  script:
    # Define image tag name
    - image="${HARBOR_CONTAINER_REGISTRY}/dynamic-environmnets-1/${DEPLOYMENT_ENVIRONMENT}/website-preview:${VERSION}"
    # Build and tag image
    - docker build -f Dockerfiles/Dockerfile --pull -t "${image}" .
    # Push image to Harbor
    - docker push "${image}"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Dockerfile
#

  • Dockerfiles/Dockerfile
FROM --platform=linux/amd64 nginx:1.18

COPY website/index.html /usr/share/nginx/html

Example Website
#

  • website/index.html
<!DOCTYPE html>
<html>

<head>
        <title>K8s Argo CD</title>
</head>

<body>
  <h1>Argo CD Dynamic Environments</h1>
  <p>Example Deployment - feature 2</p>
</body>

</html>



GitLab Helm Repository
#

File and Folder Structure
# 

# dyn-env-helm
├── helm-chart
│   ├── Chart.yaml
│   ├── templates
│   │   ├── deployment.yaml
│   │   ├── harbor-registry-secret.yaml
│   │   ├── ingress.yaml
│   │   ├── namespace.yaml
│   │   └── service.yaml
│   └── values.yaml
└── README.md

Add Deploy Key
#

Add the previously created public SSH key, so that Argo CD can access the Helm chart repository:

  • Go to: (Settings) > “Repository” > “Deploy keys”

  • Click “Add new key”

  • Define a title like “argocd_deploy”

  • Paste the public SSH key into the “Key” section

  • Disable “Grant write permissions to this key”

  • Click “Add key”


Helm Chart
#

values.yaml
#

  • helm-chart/values.yaml
example_app:
  repository: ""
  tag: main_0
  ingress_domain: dynamic-environments.jklug.work
  container_target_port: 80
imagePullSecrets:
  - harbor-registry-secret # The Kubernetes secret name
harbor:
  server: harbor.jklug.work
  username: ""
  password: ""

Chart.yaml
#

  • helm-chart/Chart.yaml
apiVersion: v2
name: Dynamic Environments
description: Helm chart for Dynamic Environments Playground
type: application
version: "0"
appVersion: "0"
message: |
    first commit

namespace.yaml
#

  • templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

deployment.yaml
#

  • templates/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamic-environments-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamic-environments
  template:
    metadata:
      labels:
        app: dynamic-environments
    spec:
      imagePullSecrets:
        - name: "{{ .Values.imagePullSecrets | first }}"
      containers:
        - name: dynamic-environments
          image: "{{ .Values.example_app.repository }}:{{ .Values.example_app.tag }}"
          ports:
            - containerPort: {{ .Values.example_app.container_target_port | int }}
          imagePullPolicy: IfNotPresent

service.yaml
#

  • templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: dynamic-environments-service
spec:
  type: ClusterIP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 900
  selector:
    app: dynamic-environments
  ports:
    - name: http
      protocol: TCP
      port: {{ .Values.example_app.container_target_port | int }}
      targetPort: {{ .Values.example_app.container_target_port | int }}

ingress.yaml
#

  • templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dynamic-environments-ingress
  annotations:
    #cert-manager.io/cluster-issuer: cluster-issuer
    nginx.ingress.kubernetes.io/ssl-redirect: "false"  # Set to true in prod
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx-metallb
  #tls:
    #- hosts:
        #- "{{ .Values.example_app.ingress_domain }}"
      #secretName: "example-app-tls-secret"
  rules:
    - host: "{{ .Values.example_app.ingress_domain }}"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: dynamic-environments-service
                port:
                  number: {{ .Values.example_app.container_target_port | int }}

harbor-registry-secret.yaml
#

Add the Harbor project credentials to the Kubernetes secret:

# Create the Harbor authentication string
REGISTRY='harbor.jklug.work'
USER='robot$dynamic-environmnets-1+dynamic-env-admin'
PASS='RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce'

AUTH="$(printf '%s:%s' "$USER" "$PASS" | base64 -w0)"

JSON="$(cat <<EOF
{"auths":{"$REGISTRY":{"username":"$USER","password":"$PASS","auth":"$AUTH"}}}
EOF
)"

printf '%s' "$JSON" | base64 -w0
echo

# Shell output:
eyJhdXRocyI6eyJoYXJib3IuamtsdWcud29yayI6eyJ1c2VybmFtZSI6InJvYm90JGR5bmFtaWMtZW52aXJvbm1uZXRzLTErZHluYW1pYy1lbnYtYWRtaW4iLCJwYXNzd29yZCI6IlJYT2FheDZGTDdXdFg5V0Y3ZVhLUnlkUWxFMFIyVWNlIiwiYXV0aCI6ImNtOWliM1FrWkhsdVlXMXBZeTFsYm5acGNtOXViVzVsZEhNdE1TdGtlVzVoYldsakxXVnVkaTFoWkcxcGJqcFNXRTloWVhnMlJrdzNWM1JZT1ZkR04yVllTMUo1WkZGc1JUQlNNbFZqWlE9PSJ9fX0=

# Verify / decode
echo 'eyJhdXRocyI6eyJoYXJib3IuamtsdWcud29yayI6eyJ1c2VybmFtZSI6InJvYm90JGR5bmFtaWMtZW52aXJvbm1uZXRzLTErZHluYW1pYy1lbnYtYWRtaW4iLCJwYXNzd29yZCI6IlJYT2FheDZGTDdXdFg5V0Y3ZVhLUnlkUWxFMFIyVWNlIiwiYXV0aCI6ImNtOWliM1FrWkhsdVlXMXBZeTFsYm5acGNtOXViVzVsZEhNdE1TdGtlVzVoYldsakxXVnVkaTFoWkcxcGJqcFNXRTloWVhnMlJrdzNWM1JZT1ZkR04yVllTMUo1WkZGc1JUQlNNbFZqWlE9PSJ9fX0=' | base64 -d

# Shell output:
{"auths":{"harbor.example.com":{"username":"robot$dynamic-environmnets-1+dynamic-env-admin","password":"RXOaax6FL7WtX9WF7eXKRydQlE0R2Uce","auth":"cm9ib3QkZHluYW1pYy1lbnZpcm9ubW5ldHMtMStkeW5hbWljLWVudi1hZG1pbjpSWE9hYXg2Rkw3V3RYOVdGN2VYS1J5ZFFsRTBSMlVjZQ=="}}}
  • harbor-registry-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: harbor-registry-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJoYXJib3IuamtsdWcud29yayI6eyJ1c2VybmFtZSI6InJvYm90JGR5bmFtaWMtZW52aXJvbm1uZXRzLTErZHluYW1pYy1lbnYtYWRtaW4iLCJwYXNzd29yZCI6IlJYT2FheDZGTDdXdFg5V0Y3ZVhLUnlkUWxFMFIyVWNlIiwiYXV0aCI6ImNtOWliM1FrWkhsdVlXMXBZeTFsYm5acGNtOXViVzVsZEhNdE1TdGtlVzVoYldsakxXVnVkaTFoWkcxcGJqcFNXRTloWVhnMlJrdzNWM1JZT1ZkR04yVllTMUo1WkZGc1JUQlNNbFZqWlE9PSJ9fX0=



Ansible Playbook
#

Apply ApplicationSet
# 

---
# Deploy Dynamic Environments ApplicationSet - Playground
- name: Deploy Dynamic Environments ApplicationSet - Playground
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    # Vault Secrets
    argocd_repository_keys: "{{ lookup('community.hashi_vault.vault_kv2_get', 'services/argocd/gitlab_deploykey', engine_mount_point='homelab_prod').secret }}"
    # Vault Variables
    argocd_repository_private_key: "{{ argocd_repository_keys.private_key }}"  # Key to access GitLab Helm chart repository
    # GitLab Helm Chart Repository
    gitlab_helm_repository: "git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git"  # Define GitLab Helm chart repository
    gitlab_repository_name: "dynamic-environments-helm"  # Just a unique name for the GitLab repository in Argo CD
    # GitLab Code Repository
    project_access_token: "glpat-49lc27dJBfr6zYKYH9egQm86MQp1OjYH.01.0w0z21y50"
    # Harbor Credentials
    harbor_user: "robot$dynamic-environmnets-1+dynamic-env-admin"
    harbor_pw: "XOaax6FL7WtX9WF7eXKRydQlE0R2Uce"
    # Argo CD Variables
    argocd_namespace: "argocd"  # Argo CD default namespace

  tasks:
    - name: Add GitLab Helm Repository via Argo CD Repository Secret
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: "{{ gitlab_repository_name }}"
            namespace: "{{ argocd_namespace }}"
            labels:
              argocd.argoproj.io/secret-type: repository
          type: Opaque
          stringData:
            url: "{{ gitlab_helm_repository }}"
            sshPrivateKey: "{{ argocd_repository_private_key }}"

    - name: Add GitLab Code Repository Secret
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: dynamic-environments-api
            namespace: "{{ argocd_namespace }}"
          type: Opaque
          stringData:
            token: "{{ project_access_token }}"  # scope read_api, role developer

    - name: Create Argo CD AppProject
      kubernetes.core.k8s:
        state: present
        namespace: argocd
        definition: |
          apiVersion: argoproj.io/v1alpha1
          kind: AppProject
          metadata:
            name: dynamic-environments-playground
            namespace: argocd
          spec:
            description: Playground for Dynamic Preview Environments
            sourceRepos:
              - '{{ gitlab_helm_repository }}'
            destinations:
              - namespace: '*'  # Allow deployment to any namespace
                server: '*'  # Allow deployment to any cluster
            clusterResourceWhitelist:
              - group: ''  # Core API group
                kind: Namespace          

    - name: Apply Argo CD ApplicationSet
      kubernetes.core.k8s:
        state: present
        namespace: argocd
        definition: |
          apiVersion: argoproj.io/v1alpha1
          kind: ApplicationSet
          metadata:
            name: dynamic-environments
            namespace: argocd
          spec:
            goTemplate: true  # Enables variable references
            generators:
              - pullRequest:
                  requeueAfterSeconds: 30  # Poll for changes
                  gitlab:
                    api: https://gitlab.jklug.work
                    project: "3"  # Code repository project ID
                    tokenRef:
                      secretName: dynamic-environments-api  # Kubernetes secret name
                      key: token
                    labels: ["preview"]
                    pullRequestState: opened
            template:
              metadata:
                name: "{% raw %}review-app-{{ .number }}{% endraw %}"
              spec:
                project: default
                source:
                  repoURL: '{{ gitlab_helm_repository }}'  # Helm repository URL
                  targetRevision: HEAD  # Or a stable branch/tag
                  path: helm-chart  # Path to Helm chart
                  helm:
                    parameters:
                      - name: example_app.repository
                        value: 'harbor.jklug.work/dynamic-environmnets-1/preview/website-preview'
                      - name: example_app.tag
                        value: "{% raw %}{{ .number }}_{{ .head_short_sha }}{% endraw %}"
                      - name: example_app.ingress_domain
                        value: "{% raw %}{{ .branch_slug }}-{{ .number }}.dynamic-environments.jklug.work{% endraw %}"
                      - name: harbor.username
                        value: '{{ harbor_user }}'
                        forceString: true
                      - name: harbor.password
                        value: '{{ harbor_pw }}'
                        forceString: true
                destination:
                  server: https://kubernetes.default.svc
                  namespace: "{% raw %}argocdplayground-{{ .branch_slug }}-{{ .number }}{% endraw %}"
                syncPolicy:
                  automated:
                    prune: true
                    selfHeal: true
                  syncOptions:
                    - PruneLast=true  # Prune on deletion          

# Run Ansible playbook: 
ansible-playbook playbooks/argocd_dynamic_environments.yml -i inventory



GitLab Code Repository
#

Triger Dynamic Deployment
# 

# Create a newfeature branch
git checkout -b feature/feature102

# Add a change
vi website/index.html
git add website/index.html

# Commit the change
git commit -m "feature102"

# Push the change
git push -u origin feature/feature102

Useful Git Commands:

# Delete the local feature branch
git branch -d feature/feature101

# Delete the remote feature branch
git push origin --delete feature/feature101

Create Merge Request
#

Create a merge request with the label “preview”:

Add the “preview” label:

Verify the GitLab pipeline ran throigh:



Argo CD
#

Verify Application
#

After the Merge request was created, Argo CD creates an application:

After the merge request was merged or closed, Argo CD automatically removes the application.



Arg oCD CLI: Verify Resources
#

CLI Login
# 

# Login to Argo CD instance
argocd login argocd.jklug.work --grpc-web

# Default user
admin

List GitLab Repositories
#

The GitLab Helm chart repository is connected via the dynamic-environments-helm Kubernetes secret, to remove the GitLab repository, delete the Kubernetes secret.

# List GitLab repositories
argocd repo list

# Shell output:
TYPE  NAME  REPO                                                                           INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git         git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git  false     false  false  false  Successful

List Project
# 

# List Argo CD projects
argocd proj list

# Shell output:
default                                                                       *,*           *                                                                              */*                         <none>                        <none>          disabled            <none>
dynamic-environments-playground  Playground for Dynamic Preview Environments  *,*           git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git  /Namespace                  <none>                        <none>          disabled            <none>

# Delete project
argocd proj delete dynamic-environments-playground

List ApplicationSet
# 

# List Argo CD ApplicationSets
kubectl -n argocd get applicationsets

# Shell output:
NAME                   AGE
dynamic-environments   32s

# Delete Argo CD ApplicationSet
kubectl -n argocd delete applicationset dynamic-environments

List Application
# 

# Verify Argo CD Applications
argocd app list

# Shell output:
NAME                 CLUSTER                         NAMESPACE                              PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                                           PATH        TARGET
argocd/review-app-3  https://kubernetes.default.svc  argocdplayground-feature-feature102-3  default  Synced  Healthy  Auto-Prune  <none>      git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git  helm-chart  HEAD

# List application details
argocd app get argocd/review-app-3

# Shell output:
Name:               argocd/review-app-3
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          argocdplayground-feature-feature102-3
URL:                https://argocd.jklug.work/applications/review-app-3
Source:
- Repo:             git@gitlab.jklug.work:production/argocd-dynamic-environments/dyn-env-helm.git
  Target:           HEAD
  Path:             helm-chart
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to HEAD (f992673)
Health Status:      Healthy

GROUP              KIND        NAMESPACE                              NAME                                   STATUS     HEALTH   HOOK  MESSAGE
                   Namespace   argocdplayground-feature-feature102-3  argocdplayground-feature-feature102-3  Succeeded  Synced         namespace/argocdplayground-feature-feature102-3 created
                   Secret      argocdplayground-feature-feature102-3  harbor-registry-secret                 Synced                    secret/harbor-registry-secret created
                   Service     argocdplayground-feature-feature102-3  dynamic-environments-service           Synced     Healthy        service/dynamic-environments-service created
apps               Deployment  argocdplayground-feature-feature102-3  dynamic-environments-deployment        Synced     Healthy        deployment.apps/dynamic-environments-deployment created
networking.k8s.io  Ingress     argocdplayground-feature-feature102-3  dynamic-environments-ingress           Synced     Healthy        ingress.networking.k8s.io/dynamic-environments-ingress created
                   Namespace                                          argocdplayground-feature-feature102-3  Synced



Kubectl
#

Verify Kubernetes Resources
# 

# List default resources
kubectl get all -n argocdplayground-feature-feature102-3

# Shell output:
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/dynamic-environments-deployment-7c5744ccf9-xzb7g   1/1     Running   0          6m55s

NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/dynamic-environments-service   ClusterIP   10.201.224.241   <none>        80/TCP    6m55s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dynamic-environments-deployment   1/1     1            1           6m55s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/dynamic-environments-deployment-7c5744ccf9   1         1         1       6m55s

# List Ingress resource
kubectl get ingress -n argocdplayground-feature-feature102-3

# Shell output:
NAME                           CLASS           HOSTS                                                  ADDRESS          PORTS   AGE
dynamic-environments-ingress   nginx-metallb   feature-feature102-3.dynamic-environments.jklug.work   192.168.70.110   80      7m36s