Skip to main content

Loki, Grafana & Traefik - Docker Compose Stack: Deployment with GitLab CI Pipeline, Git Commands

2188 words·
Loki Grafana Traefik GitLab GitLab CI Git
Table of Contents
Loki-Stack - This article is part of a series.
Part 2: This Article

Overview
#

I’m using the following servers in this tutorial.

192.168.70.4 # Ubuntu 22.04: GitLab Server
192.168.70.5 # Ubuntu 22.04: GitLab Runner
192.168.70.11 # Debian 12: Deployment Server

My first example is a GitLab CI pipeline that deployes a Grafana instance with four Loki instances as data sources. Each Loki instance is used for a different Promtail scrapping configuration, as structured in the following tutorial: Loki Docker Compose Stack - Loki, Promtail, Grafana. Collect Logs from Docker Containers, /var/logs and Systemd-Journal

My second example also deployes a Traefik reverse Proxy for the Grafana instance. As usual I’m using a Let’s encrypt wildcard certificate for the traefik reverse proxy.


GitLab Runner
#

I’m using a bare-metal GitLab runner with the following configuration:

# Open the GitLab runner configuration file
sudo vi /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
shutdown_timeout = 0
log_level = "debug"
log_format = "text"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Untagged-Build-Runner-01"
  url = "https://gitlab.jklug.work"
  id = 1
  token = "mytoken..."
  token_obtained_at = 2024-02-03T21:35:10Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    extra_hosts = ["gitlab.jklug.work:192.168.70.4","gitlab-registry.jklug.work:192.168.70.4"]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0

Deployment Server
#

Hosts Entry
#

In order to pull the Grafana image from the GitLab registry, create an hosts entry, so that the deployment server is able the resolve the GitLab and GitLab registry domain name.

# Add an hosts entry for GitLab and the GitLab registry
192.168.70.4 gitlab.jklug.work gitlab-registry.jklug.work

Create Deployment User
#

# Create a user for the deployment
sudo adduser gitlab-deploy

# Add the deplyoment user to the Docker group
sudo usermod -aG docker gitlab-deploy

# Switch to the deployment user
sudo su gitlab-deploy

Create SSH Key
#

# CD into home directory
cd

# Create a SSH key
ssh-keygen -t rsa -b 4096

# Add the SSH key to the authorized_keys file
cat .ssh/id_rsa.pub >> .ssh/authorized_keys

# Copy the value of the private SSH key
cat .ssh/id_rsa

GitLab Variables
#

SSH Private Key
#

Go to: “Settings” > “CI/CD” > Expand “Variables”

  • Click “Add variable”

  • Create a “CI/CD Variable” with the name (Key): SSH_PRIVATE_KEY

  • Add the private SSH key as “Value”

  • Enable the flag “Expand variable reference”

# Make sure to add a paragraph / empty line after
-----END OPENSSH PRIVATE KEY-----

SSH Fingerprint
#

# Get the SSH fingerprint of the deployment server:
ssh-keyscan -t rsa -H 192.168.70.11

Shell Output:

# 192.168.70.11:22 SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
|1|MYZKyaj2T20fe7Y14FxoWbMFkio=|FNFYm6A0/6Vn8wYxol8yf494VTk= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwFnA9aNqr4hQtmNMOl4Mq6PmHO4BrdYbFQgK+L4qLjVDublCzJEKinVwI0LhEbRG/B+rq+ZTNTBzfL2WNU/+Xf3Y/IjgaXCbiNNSvDck2sHKhumQuONF9w0ucBBB9qnYic5qf3np2KAJSviVTVsyz5rgM+B3ylc9QK9hOnB8GNISI3uzsxGJdkWjJyF/kVr2gjOoEjC86vh3vtC5LlyN6a1wjkDtlaV2lMhj9i88ulg68a/4ngk6nGwsWexrRetQ9KJ8KGM02qfKjpFExbz0qA+VPPht+kBYWVh0qgaPjyIKryGc4DPKM1JcdlZ7wkVHL3JHHysOvjLRDKqgIyMziFj9gdjSM8FEV5M5OEzKnEQi64LMhfeg5Jql0QUR4Bs3NjblcecNxxjBd5TkprZJlT6DxDLFOXiwgoul/GVX9bkiwb2GO2YvB19ltdjRbFD0XNOeYa4gEupy7C5ToPVztiAxYQgQFdZoTlNTqzJ6l358e53gG/1hKhgwNFzfU+Tc=

Go to: “Settings” > “CI/CD” > Expand “Variables”

  • Click “Add variable”

  • Create a “CI/CD Variable” with the name (Key): SSH_HOST_KEY

  • Add the following “Value”:

|1|MYZKyaj2T20fe7Y14FxoWbMFkio=|FNFYm6A0/6Vn8wYxol8yf494VTk= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwFnA9aNqr4hQtmNMOl4Mq6PmHO4BrdYbFQgK+L4qLjVDublCzJEKinVwI0LhEbRG/B+rq+ZTNTBzfL2WNU/+Xf3Y/IjgaXCbiNNSvDck2sHKhumQuONF9w0ucBBB9qnYic5qf3np2KAJSviVTVsyz5rgM+B3ylc9QK9hOnB8GNISI3uzsxGJdkWjJyF/kVr2gjOoEjC86vh3vtC5LlyN6a1wjkDtlaV2lMhj9i88ulg68a/4ngk6nGwsWexrRetQ9KJ8KGM02qfKjpFExbz0qA+VPPht+kBYWVh0qgaPjyIKryGc4DPKM1JcdlZ7wkVHL3JHHysOvjLRDKqgIyMziFj9gdjSM8FEV5M5OEzKnEQi64LMhfeg5Jql0QUR4Bs3NjblcecNxxjBd5TkprZJlT6DxDLFOXiwgoul/GVX9bkiwb2GO2YvB19ltdjRbFD0XNOeYa4gEupy7C5ToPVztiAxYQgQFdZoTlNTqzJ6l358e53gG/1hKhgwNFzfU+Tc=
  • Enable the flag “Expand variable reference”

Protected Branches / Variables
#

  • When the variable flag Protect variable is enabled, the variable is only available for protected branches of the repository.

  • To protect a new branch go to:
    “Settings” > “Repository” > “Protected branches”

Deployment Folder Structure
#

Loki & Grafana
#

Create the folder structure for the Loki & Grafana deployment:

# Create folder structure for the volume mapping
sudo mkdir -p /opt/loki/{loki_data,loki_data2,loki_data3,loki_data4}

# Change ownership
sudo chown gitlab-deploy:docker /opt/loki &&
sudo chown 10001:10001 /opt/loki/{loki_data,loki_data2,loki_data3,loki_data4}

Traefik
#

For the Loki & Grafana stack with the Traefik reverse proxy, create the folder structure for the Traefik deployment:

# Create folder structure for traefik
sudo mkdir -p /opt/traefik/certs && cd /opt/traefik

# Change ownership
sudo chown gitlab-deploy:docker /opt/traefik

Traefik Docker Network
#

For the Loki & Grafana stack with the Traefik reverse proxy, it’s necessary to create an external Docker network:

# Create network used for Traefik to communicate with other Docker containers
sudo docker network create traefik

Loki & Grafana Stack
#

Repository Filestructure
#

├── .gitlab-ci.yml
├── docker-compose.tmpl
├── Dockerfile
└── grafana-datasource.yaml

Description
#

The following GitLab CI pipeline deployes the following steps:

  • It builds a Grafana container image from a Grafana-Dockerfile, that adds the grafana-datasource.yaml configuration file, that defines the 4 Loki instances as data sources.

  • After that the docker-compose.tmpl is adopted with the GitLab registry path for the Grafana image, and copied to the deployment server.

  • Last the Docker Compose stack is started on the deployment server.

CI Pipeline
#

GitLab CI: .gitlab-ci.yml

stages:
  - build
  - deploy

build:
  stage: build
  image: docker:20.10.16
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  services:
    - docker:20.10.16-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER 
                              -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest 
        --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA 
        --tag $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

deploy:
  stage: deploy
  image: alpine
  before_script:
    # Install dependencies
    - apk add gettext openssh-client
  script:
    # Create the docker-compose.yml with the path to the GitLab registry
    - envsubst < docker-compose.tmpl > docker-compose.yml
    # Start SSH agent and import private key
    - eval `ssh-agent`
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    # Add the deployment server to list of known hosts
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - touch ~/.ssh/known_hosts
    - chmod 600 ~/.ssh/known_hosts
    - echo $SSH_HOST_KEY >> ~/.ssh/known_hosts
    # Upload the docker-compose.yml file to the deployment server
    - scp docker-compose.yml gitlab-deploy@192.168.70.11:/opt/loki
    # Login to container registry and start the docker compose stack
    - ssh gitlab-deploy@192.168.70.11 "cd /opt/loki; 
        docker login -u $CI_REGISTRY_USER 
          -p $CI_REGISTRY_PASSWORD $CI_REGISTRY;
        docker compose up -d"

Docker Compose YML
#

Docker Compose file: docker-compose.tmpl

services:
  grafana:
    image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    ports:
      - 3000:3000
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    restart: unless-stopped
    networks:
      - loki

  loki:
    image: grafana/loki:latest
    ports:
      - 3101:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data:/loki
    restart: unless-stopped
    networks:
      - loki

  loki2:
    image: grafana/loki:latest
    ports:
      - 3102:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data2:/loki
    restart: unless-stopped
    networks:
      - loki

  loki3:
    image: grafana/loki:latest
    ports:
      - 3103:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data3:/loki
    restart: unless-stopped
    networks:
      - loki

  loki4:
    image: grafana/loki:latest
    ports:
      - 3104:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data4:/loki
    restart: unless-stopped
    networks:
      - loki

networks:
  loki:
    name: loki

Grafana Dockerfile
#

Dockerfile for Grafana: Dockerfile

# Use the latest official Grafana image as a base image
FROM grafana/grafana:latest

# Expose port 3000
EXPOSE 3000

# Copy the datasource configuration file
COPY ./grafana-datasource.yaml /etc/grafana/provisioning/datasources/datasources.yaml

# Specify the command to run on container start
CMD ["grafana-server"]

Grafana Datasource
#

Grafana datasource YML: grafana-datasource.yaml

apiVersion: 1

datasources:
  - name: Loki-Container
    type: loki
    access: proxy
    url: http://loki:3100
    version: 1
    editable: false
    isDefault: true

  - name: Loki-VarLog
    type: loki
    access: proxy
    url: http://loki2:3100
    version: 1
    editable: false
    isDefault: false

  - name: Loki-Systemd-Journal
    type: loki
    access: proxy
    url: http://loki3:3100
    version: 1
    editable: false
    isDefault: false

  - name: Loki-Kubernetes
    type: loki
    access: proxy
    url: http://loki4:3100
    version: 1
    editable: false
    isDefault: false

Loki, Grafana & Traefik Stack
#

Repository Filestructure
#

├── Docker-Compose
│   ├── loki_docker-compose.tmpl
│   └── traefik_docker-compose.tmpl
├── Dockerfiles
│   ├── Grafana-Dockerfile
│   └── Traefik-Dockerfile
├── grafana-datasource.yaml
├── tls-configuration.yaml
└── Traefik-Certs
    ├── fullchain.pem
    └── privkey.pem

Description
#

The following GitLab CI pipeline deployes the following steps:

  • It builds a Grafana container image from a Grafana-Dockerfile, that adds the grafana-datasource.yaml configuration file, that defines the 4 Loki instances as data sources.

  • It builds a Traefik container image from a Traefik-Dockerfile, that adds the tls-configuration.yaml configuration file and adds the TLS certificates from the Traefik-Certs directory.

  • After that the loki_docker-compose.tmpl and traefik_docker-compose.tmpl are adopted with the GitLab registry patsh for the images, and copied to the deployment server.

  • Last the Docker Compose stacks are started on the deployment server.

GitLab CI Pipeline
#

GitLab CI: .gitlab-ci.yml

stages:
  - build-grafana
  - build-traefik
  - deploy

variables:
  GRAFANA_IMAGE: $CI_REGISTRY_IMAGE/grafana:$CI_COMMIT_SHA
  TRAEFIK_IMAGE: $CI_REGISTRY_IMAGE/traefik:$CI_COMMIT_SHA

build-grafana:
  stage: build-grafana
  image: docker:20.10.16
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  services:
    - docker:20.10.16-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker pull $CI_REGISTRY_IMAGE/grafana:latest || true
    - docker build -f Dockerfiles/Grafana-Dockerfile --cache-from $CI_REGISTRY_IMAGE/grafana:latest 
      --tag $GRAFANA_IMAGE --tag $CI_REGISTRY_IMAGE/grafana:latest .
    - docker push $GRAFANA_IMAGE
    - docker push $CI_REGISTRY_IMAGE/grafana:latest

build-traefik:
  stage: build-traefik
  image: docker:20.10.16
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  services:
    - docker:20.10.16-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker pull $CI_REGISTRY_IMAGE/traefik:latest || true
    - docker build -f Dockerfiles/Traefik-Dockerfile --cache-from $CI_REGISTRY_IMAGE/traefik:latest 
      --tag $TRAEFIK_IMAGE --tag $CI_REGISTRY_IMAGE/traefik:latest .
    - docker push $TRAEFIK_IMAGE
    - docker push $CI_REGISTRY_IMAGE/traefik:latest

deploy:
  stage: deploy
  image: alpine
  before_script:
    - apk add gettext openssh-client
    - eval `ssh-agent`
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - touch ~/.ssh/known_hosts
    - chmod 600 ~/.ssh/known_hosts
    - echo $SSH_HOST_KEY >> ~/.ssh/known_hosts
  script:
    - envsubst < Docker-Compose/loki_docker-compose.tmpl > docker-compose-grafana.yml
    - envsubst < Docker-Compose/traefik_docker-compose.tmpl > docker-compose-traefik.yml
    - scp docker-compose-grafana.yml gitlab-deploy@192.168.70.11:/opt/loki
    - scp docker-compose-traefik.yml gitlab-deploy@192.168.70.11:/opt/traefik
    - ssh gitlab-deploy@192.168.70.11 "cd /opt/loki; docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY; docker compose -f docker-compose-grafana.yml up -d"
    - ssh gitlab-deploy@192.168.70.11 "cd /opt/traefik; docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY; docker compose -f docker-compose-traefik.yml up -d"

Loki & Grafana Stack
#

Docker Compose YML
#

Docker Compose file: Docker-Compose/loki_docker-compose.tmpl

services:
  grafana:
    image: ${GRAFANA_IMAGE}
    ports:
      - 3000:3000
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik"
      - "traefik.http.services.loki.loadbalancer.server.port=3000"
      # HTTPS Router
      - "traefik.http.routers.loki.entrypoints=websecure"
      - "traefik.http.routers.loki.tls=true"
      - "traefik.http.routers.loki.rule=Host(`loki.jklug.work`)"
      # HTTP Router
      - "traefik.http.routers.loki-http.rule=Host(`loki.jklug.work`)"
      - "traefik.http.routers.loki-http.entrypoints=web"
      - "traefik.http.routers.loki-http.middlewares=redirect-to-https"
      # Middleware for HTTP to HTTPS redirection
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    networks:
      - loki
      - traefik

  loki:
    image: grafana/loki:latest
    ports:
      - 3101:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data:/loki
    restart: unless-stopped
    labels:
      - "traefik.enable=false"
    networks:
      - loki

  loki2:
    image: grafana/loki:latest
    ports:
      - 3102:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data2:/loki
    restart: unless-stopped
    labels:
      - "traefik.enable=false"
    networks:
      - loki

  loki3:
    image: grafana/loki:latest
    ports:
      - 3103:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data3:/loki
    restart: unless-stopped
    labels:
      - "traefik.enable=false"
    networks:
      - loki

  loki4:
    image: grafana/loki:latest
    ports:
      - 3104:3100
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki_data4:/loki
    restart: unless-stopped
    labels:
      - "traefik.enable=false"
    networks:
      - loki

networks:
  loki:
    name: loki
  traefik:
    external: true

Dockerfile
#

Dockerfile for Grafana: Dockerfiles/Grafana-Dockerfile

# Use the latest official Grafana image as a base
FROM grafana/grafana:latest

# Expose port 3000
EXPOSE 3000

# Copy the datasource configuration file
COPY ./grafana-datasource.yaml /etc/grafana/provisioning/datasources/datasources.yaml

# Specify the command to run on container start
CMD ["grafana-server"]

Grafana Datasource
#

Grafana datasource YML: grafana-datasource.yaml

apiVersion: 1

datasources:
  - name: Loki-Container
    type: loki
    access: proxy
    url: http://loki:3100
    version: 1
    editable: false
    isDefault: true

  - name: Loki-VarLog
    type: loki
    access: proxy
    url: http://loki2:3100
    version: 1
    editable: false
    isDefault: false

  - name: Loki-Systemd-Journal
    type: loki
    access: proxy
    url: http://loki3:3100
    version: 1
    editable: false
    isDefault: false

  - name: Loki-Kubernetes
    type: loki
    access: proxy
    url: http://loki4:3100
    version: 1
    editable: false
    isDefault: false

Traefik
#

Docker Compose YML
#

Docker Compose file: Docker-Compose/traefik_docker-compose.tmpl

version: '3'
services:
  reverse-proxy:
    image: ${TRAEFIK_IMAGE}
    container_name: Traefik-Reverse-Proxy
    restart: unless-stopped
    command:
      - --api.insecure=true
      - --providers.docker
      - --providers.file.directory=/etc/traefik/dynamic
      - --entryPoints.web.address=:80
      - --entryPoints.websecure.address=:443
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - traefik

networks:
  traefik:
    external: true

Traefik Dockerfile
#

Dockerfile for Traefik: Dockerfiles/Traefik-Dockerfile

# Use the latest Traefik image as base
FROM traefik:v2.10

# Expose ports 80,443,8080
EXPOSE 80
EXPOSE 443
EXPOSE 8080

# Copy the tls-configuration configuration file
COPY ./tls-configuration.yaml /etc/traefik/dynamic/tls-configuration.yaml
# Copy the TLS certificates
COPY ./Traefik-Certs/ /etc/certs/

# Specify the command to run on container start
CMD ["traefik", "--configFile=/etc/traefik/traefik.yml"]

Traefik Dynamic Configuration
#

Traefik tls-configuration YML: tls-configuration.yaml

tls:
  certificates:
    - certFile: /etc/certs/fullchain.pem
      keyFile: /etc/certs/privkey.pem

Git PowerShell Commands
#

Install Git with Winget
#

  • Open PowerShell as Administrator
# Install Git
winget install Git.Git
  • Close and open a new PowerShell
# Check Git version
git --version

Prerequisites
#

Set the username and email address that will be associated with any commits you create across all Git repositories on your computer.

# Define your name
git config --global user.name "Your Name"

# Define youremail
git config --global user.email "you@example.com"

Repository
#

Clone the repository
#

# Change directory: Open a directory where the GitLab repository should be saved
cd my-git-projects

# Clone the repository
git clone git@gitlab.jklug.work:root/loki-grafana-docker-compose-stack.git

# CD into the repository
cd .\loki-grafana-docker-compose-stack\

Create / Switch Branch
#

# List available branches: Local
git branch

# List available branches: Remote
git branch -r

# List available branches: Local & remote
git branch -a
# Create main branch & switch into main branch (The main branch should already be existing)
git switch --create main

# Switch to main branch
git switch main

Commit File or Folder
#

# Stages the "foldername" folder and its contents for the next commit
git add .\foldername\

# Stage all new and modified files in current folder
git add .


# Commit the changes
git commit -m "Added folder"


# Push the commit to the current branch of the remote repository on GitLab
git push

# Push the commit to the main branch of the remote repository on GitLab
git push origin main

Fetch & Pull Changes
#

# Verify the current branch
git branch

# Shell output:
* main
# Fetch the changes to the local branch
git fetch

# Fetch the changes to the local branch: All branches
git fetch --all
  • Git Fetch Download but not merge the changes from the remote repository.
# Pull the changes to the local branch
git pull
  • Git Pull Git first fetches the updates from the remote repository (the same as git fetch) and then automatically tries to merge the latest version of the branch from the remote into the current local branch.

Checkout Branch
#

# Checkout specific remote branch "loki-grafana-main" to the local repository
git checkout -b loki-grafana-main origin/loki-grafana-main

Links #

# Loki & Grafana stack on GitHub
https://github.com/jueklu/loki-grafana-stack

# Loki, Grafana & Traefik stack on GitHub
https://github.com/jueklu/loki-grafana-traefik-stack
Loki-Stack - This article is part of a series.
Part 2: This Article