Overview #
I’m using the following servers in this tutorial.
192.168.70.4 # Ubuntu 22.04: GitLab Server
192.168.70.5 # Ubuntu 22.04: GitLab Runner
192.168.70.11 # Debian 12: Deployment Server
My first example is a GitLab CI pipeline that deployes a Grafana instance with four Loki instances as data sources. Each Loki instance
is used for a different Promtail scrapping configuration, as structured in the following tutorial:
Loki Docker Compose Stack - Loki, Promtail, Grafana. Collect Logs from Docker Containers, /var/logs and Systemd-Journal
My second example also deployes a Traefik reverse Proxy for the Grafana instance. As usual I’m using a Let’s encrypt wildcard certificate for the traefik reverse proxy.
GitLab Runner #
I’m using a bare-metal GitLab runner with the following configuration:
# Open the GitLab runner configuration file
sudo vi /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
shutdown_timeout = 0
log_level = "debug"
log_format = "text"
[session_server]
session_timeout = 1800
[[runners]]
name = "Untagged-Build-Runner-01"
url = "https://gitlab.jklug.work"
id = 1
token = "mytoken..."
token_obtained_at = 2024-02-03T21:35:10Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.docker]
extra_hosts = ["gitlab.jklug.work:192.168.70.4","gitlab-registry.jklug.work:192.168.70.4"]
tls_verify = false
image = "docker:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
network_mtu = 0
Deployment Server #
Hosts Entry #
In order to pull the Grafana image from the GitLab registry, create an hosts entry, so that the deployment server is able the resolve the GitLab and GitLab registry domain name.
# Add an hosts entry for GitLab and the GitLab registry
192.168.70.4 gitlab.jklug.work gitlab-registry.jklug.work
Create Deployment User #
# Create a user for the deployment
sudo adduser gitlab-deploy
# Add the deplyoment user to the Docker group
sudo usermod -aG docker gitlab-deploy
# Switch to the deployment user
sudo su gitlab-deploy
Create SSH Key #
# CD into home directory
cd
# Create a SSH key
ssh-keygen -t rsa -b 4096
# Add the SSH key to the authorized_keys file
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
# Copy the value of the private SSH key
cat .ssh/id_rsa
GitLab Variables #
SSH Private Key #
Go to: “Settings” > “CI/CD” > Expand “Variables”
-
Click “Add variable”
-
Create a “CI/CD Variable” with the name (Key):
SSH_PRIVATE_KEY
-
Add the private SSH key as “Value”
-
Enable the flag “Expand variable reference”
# Make sure to add a paragraph / empty line after
-----END OPENSSH PRIVATE KEY-----
SSH Fingerprint #
# Get the SSH fingerprint of the deployment server:
ssh-keyscan -t rsa -H 192.168.70.11
Shell Output:
# 192.168.70.11:22 SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
|1|MYZKyaj2T20fe7Y14FxoWbMFkio=|FNFYm6A0/6Vn8wYxol8yf494VTk= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwFnA9aNqr4hQtmNMOl4Mq6PmHO4BrdYbFQgK+L4qLjVDublCzJEKinVwI0LhEbRG/B+rq+ZTNTBzfL2WNU/+Xf3Y/IjgaXCbiNNSvDck2sHKhumQuONF9w0ucBBB9qnYic5qf3np2KAJSviVTVsyz5rgM+B3ylc9QK9hOnB8GNISI3uzsxGJdkWjJyF/kVr2gjOoEjC86vh3vtC5LlyN6a1wjkDtlaV2lMhj9i88ulg68a/4ngk6nGwsWexrRetQ9KJ8KGM02qfKjpFExbz0qA+VPPht+kBYWVh0qgaPjyIKryGc4DPKM1JcdlZ7wkVHL3JHHysOvjLRDKqgIyMziFj9gdjSM8FEV5M5OEzKnEQi64LMhfeg5Jql0QUR4Bs3NjblcecNxxjBd5TkprZJlT6DxDLFOXiwgoul/GVX9bkiwb2GO2YvB19ltdjRbFD0XNOeYa4gEupy7C5ToPVztiAxYQgQFdZoTlNTqzJ6l358e53gG/1hKhgwNFzfU+Tc=
Go to: “Settings” > “CI/CD” > Expand “Variables”
-
Click “Add variable”
-
Create a “CI/CD Variable” with the name (Key):
SSH_HOST_KEY
-
Add the following “Value”:
|1|MYZKyaj2T20fe7Y14FxoWbMFkio=|FNFYm6A0/6Vn8wYxol8yf494VTk= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwFnA9aNqr4hQtmNMOl4Mq6PmHO4BrdYbFQgK+L4qLjVDublCzJEKinVwI0LhEbRG/B+rq+ZTNTBzfL2WNU/+Xf3Y/IjgaXCbiNNSvDck2sHKhumQuONF9w0ucBBB9qnYic5qf3np2KAJSviVTVsyz5rgM+B3ylc9QK9hOnB8GNISI3uzsxGJdkWjJyF/kVr2gjOoEjC86vh3vtC5LlyN6a1wjkDtlaV2lMhj9i88ulg68a/4ngk6nGwsWexrRetQ9KJ8KGM02qfKjpFExbz0qA+VPPht+kBYWVh0qgaPjyIKryGc4DPKM1JcdlZ7wkVHL3JHHysOvjLRDKqgIyMziFj9gdjSM8FEV5M5OEzKnEQi64LMhfeg5Jql0QUR4Bs3NjblcecNxxjBd5TkprZJlT6DxDLFOXiwgoul/GVX9bkiwb2GO2YvB19ltdjRbFD0XNOeYa4gEupy7C5ToPVztiAxYQgQFdZoTlNTqzJ6l358e53gG/1hKhgwNFzfU+Tc=
- Enable the flag “Expand variable reference”
Protected Branches / Variables #
-
When the variable flag
Protect variable
is enabled, the variable is only available for protected branches of the repository. -
To protect a new branch go to:
“Settings” > “Repository” > “Protected branches”
Deployment Folder Structure #
Loki & Grafana #
Create the folder structure for the Loki & Grafana deployment:
# Create folder structure for the volume mapping
sudo mkdir -p /opt/loki/{loki_data,loki_data2,loki_data3,loki_data4}
# Change ownership
sudo chown gitlab-deploy:docker /opt/loki &&
sudo chown 10001:10001 /opt/loki/{loki_data,loki_data2,loki_data3,loki_data4}
Traefik #
For the Loki & Grafana stack with the Traefik reverse proxy, create the folder structure for the Traefik deployment:
# Create folder structure for traefik
sudo mkdir -p /opt/traefik/certs && cd /opt/traefik
# Change ownership
sudo chown gitlab-deploy:docker /opt/traefik
Traefik Docker Network #
For the Loki & Grafana stack with the Traefik reverse proxy, it’s necessary to create an external Docker network:
# Create network used for Traefik to communicate with other Docker containers
sudo docker network create traefik
Loki & Grafana Stack #
Repository Filestructure #
├── .gitlab-ci.yml
├── docker-compose.tmpl
├── Dockerfile
└── grafana-datasource.yaml
Description #
The following GitLab CI pipeline deployes the following steps:
-
It builds a Grafana container image from a
Grafana-Dockerfile
, that adds thegrafana-datasource.yaml
configuration file, that defines the 4 Loki instances as data sources. -
After that the docker-compose.tmpl is adopted with the GitLab registry path for the Grafana image, and copied to the deployment server.
-
Last the Docker Compose stack is started on the deployment server.
CI Pipeline #
GitLab CI: .gitlab-ci.yml
stages:
- build
- deploy
build:
stage: build
image: docker:20.10.16
variables:
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
services:
- docker:20.10.16-dind
before_script:
- docker login -u $CI_REGISTRY_USER
-p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker pull $CI_REGISTRY_IMAGE:latest || true
- docker build --cache-from $CI_REGISTRY_IMAGE:latest
--tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
--tag $CI_REGISTRY_IMAGE:latest .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
deploy:
stage: deploy
image: alpine
before_script:
# Install dependencies
- apk add gettext openssh-client
script:
# Create the docker-compose.yml with the path to the GitLab registry
- envsubst < docker-compose.tmpl > docker-compose.yml
# Start SSH agent and import private key
- eval `ssh-agent`
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# Add the deployment server to list of known hosts
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- touch ~/.ssh/known_hosts
- chmod 600 ~/.ssh/known_hosts
- echo $SSH_HOST_KEY >> ~/.ssh/known_hosts
# Upload the docker-compose.yml file to the deployment server
- scp docker-compose.yml gitlab-deploy@192.168.70.11:/opt/loki
# Login to container registry and start the docker compose stack
- ssh gitlab-deploy@192.168.70.11 "cd /opt/loki;
docker login -u $CI_REGISTRY_USER
-p $CI_REGISTRY_PASSWORD $CI_REGISTRY;
docker compose up -d"
Docker Compose YML #
Docker Compose file: docker-compose.tmpl
services:
grafana:
image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
ports:
- 3000:3000
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
restart: unless-stopped
networks:
- loki
loki:
image: grafana/loki:latest
ports:
- 3101:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data:/loki
restart: unless-stopped
networks:
- loki
loki2:
image: grafana/loki:latest
ports:
- 3102:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data2:/loki
restart: unless-stopped
networks:
- loki
loki3:
image: grafana/loki:latest
ports:
- 3103:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data3:/loki
restart: unless-stopped
networks:
- loki
loki4:
image: grafana/loki:latest
ports:
- 3104:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data4:/loki
restart: unless-stopped
networks:
- loki
networks:
loki:
name: loki
Grafana Dockerfile #
Dockerfile for Grafana: Dockerfile
# Use the latest official Grafana image as a base image
FROM grafana/grafana:latest
# Expose port 3000
EXPOSE 3000
# Copy the datasource configuration file
COPY ./grafana-datasource.yaml /etc/grafana/provisioning/datasources/datasources.yaml
# Specify the command to run on container start
CMD ["grafana-server"]
Grafana Datasource #
Grafana datasource YML: grafana-datasource.yaml
apiVersion: 1
datasources:
- name: Loki-Container
type: loki
access: proxy
url: http://loki:3100
version: 1
editable: false
isDefault: true
- name: Loki-VarLog
type: loki
access: proxy
url: http://loki2:3100
version: 1
editable: false
isDefault: false
- name: Loki-Systemd-Journal
type: loki
access: proxy
url: http://loki3:3100
version: 1
editable: false
isDefault: false
- name: Loki-Kubernetes
type: loki
access: proxy
url: http://loki4:3100
version: 1
editable: false
isDefault: false
Loki, Grafana & Traefik Stack #
Repository Filestructure #
├── Docker-Compose
│ ├── loki_docker-compose.tmpl
│ └── traefik_docker-compose.tmpl
├── Dockerfiles
│ ├── Grafana-Dockerfile
│ └── Traefik-Dockerfile
├── grafana-datasource.yaml
├── tls-configuration.yaml
└── Traefik-Certs
├── fullchain.pem
└── privkey.pem
Description #
The following GitLab CI pipeline deployes the following steps:
-
It builds a Grafana container image from a
Grafana-Dockerfile
, that adds thegrafana-datasource.yaml
configuration file, that defines the 4 Loki instances as data sources. -
It builds a Traefik container image from a
Traefik-Dockerfile
, that adds thetls-configuration.yaml
configuration file and adds the TLS certificates from theTraefik-Certs
directory. -
After that the
loki_docker-compose.tmpl
andtraefik_docker-compose.tmpl
are adopted with the GitLab registry patsh for the images, and copied to the deployment server. -
Last the Docker Compose stacks are started on the deployment server.
GitLab CI Pipeline #
GitLab CI: .gitlab-ci.yml
stages:
- build-grafana
- build-traefik
- deploy
variables:
GRAFANA_IMAGE: $CI_REGISTRY_IMAGE/grafana:$CI_COMMIT_SHA
TRAEFIK_IMAGE: $CI_REGISTRY_IMAGE/traefik:$CI_COMMIT_SHA
build-grafana:
stage: build-grafana
image: docker:20.10.16
variables:
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
services:
- docker:20.10.16-dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker pull $CI_REGISTRY_IMAGE/grafana:latest || true
- docker build -f Dockerfiles/Grafana-Dockerfile --cache-from $CI_REGISTRY_IMAGE/grafana:latest
--tag $GRAFANA_IMAGE --tag $CI_REGISTRY_IMAGE/grafana:latest .
- docker push $GRAFANA_IMAGE
- docker push $CI_REGISTRY_IMAGE/grafana:latest
build-traefik:
stage: build-traefik
image: docker:20.10.16
variables:
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
services:
- docker:20.10.16-dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker pull $CI_REGISTRY_IMAGE/traefik:latest || true
- docker build -f Dockerfiles/Traefik-Dockerfile --cache-from $CI_REGISTRY_IMAGE/traefik:latest
--tag $TRAEFIK_IMAGE --tag $CI_REGISTRY_IMAGE/traefik:latest .
- docker push $TRAEFIK_IMAGE
- docker push $CI_REGISTRY_IMAGE/traefik:latest
deploy:
stage: deploy
image: alpine
before_script:
- apk add gettext openssh-client
- eval `ssh-agent`
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- touch ~/.ssh/known_hosts
- chmod 600 ~/.ssh/known_hosts
- echo $SSH_HOST_KEY >> ~/.ssh/known_hosts
script:
- envsubst < Docker-Compose/loki_docker-compose.tmpl > docker-compose-grafana.yml
- envsubst < Docker-Compose/traefik_docker-compose.tmpl > docker-compose-traefik.yml
- scp docker-compose-grafana.yml gitlab-deploy@192.168.70.11:/opt/loki
- scp docker-compose-traefik.yml gitlab-deploy@192.168.70.11:/opt/traefik
- ssh gitlab-deploy@192.168.70.11 "cd /opt/loki; docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY; docker compose -f docker-compose-grafana.yml up -d"
- ssh gitlab-deploy@192.168.70.11 "cd /opt/traefik; docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY; docker compose -f docker-compose-traefik.yml up -d"
Loki & Grafana Stack #
Docker Compose YML #
Docker Compose file: Docker-Compose/loki_docker-compose.tmpl
services:
grafana:
image: ${GRAFANA_IMAGE}
ports:
- 3000:3000
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik"
- "traefik.http.services.loki.loadbalancer.server.port=3000"
# HTTPS Router
- "traefik.http.routers.loki.entrypoints=websecure"
- "traefik.http.routers.loki.tls=true"
- "traefik.http.routers.loki.rule=Host(`loki.jklug.work`)"
# HTTP Router
- "traefik.http.routers.loki-http.rule=Host(`loki.jklug.work`)"
- "traefik.http.routers.loki-http.entrypoints=web"
- "traefik.http.routers.loki-http.middlewares=redirect-to-https"
# Middleware for HTTP to HTTPS redirection
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
networks:
- loki
- traefik
loki:
image: grafana/loki:latest
ports:
- 3101:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data:/loki
restart: unless-stopped
labels:
- "traefik.enable=false"
networks:
- loki
loki2:
image: grafana/loki:latest
ports:
- 3102:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data2:/loki
restart: unless-stopped
labels:
- "traefik.enable=false"
networks:
- loki
loki3:
image: grafana/loki:latest
ports:
- 3103:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data3:/loki
restart: unless-stopped
labels:
- "traefik.enable=false"
networks:
- loki
loki4:
image: grafana/loki:latest
ports:
- 3104:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki_data4:/loki
restart: unless-stopped
labels:
- "traefik.enable=false"
networks:
- loki
networks:
loki:
name: loki
traefik:
external: true
Dockerfile #
Dockerfile for Grafana: Dockerfiles/Grafana-Dockerfile
# Use the latest official Grafana image as a base
FROM grafana/grafana:latest
# Expose port 3000
EXPOSE 3000
# Copy the datasource configuration file
COPY ./grafana-datasource.yaml /etc/grafana/provisioning/datasources/datasources.yaml
# Specify the command to run on container start
CMD ["grafana-server"]
Grafana Datasource #
Grafana datasource YML: grafana-datasource.yaml
apiVersion: 1
datasources:
- name: Loki-Container
type: loki
access: proxy
url: http://loki:3100
version: 1
editable: false
isDefault: true
- name: Loki-VarLog
type: loki
access: proxy
url: http://loki2:3100
version: 1
editable: false
isDefault: false
- name: Loki-Systemd-Journal
type: loki
access: proxy
url: http://loki3:3100
version: 1
editable: false
isDefault: false
- name: Loki-Kubernetes
type: loki
access: proxy
url: http://loki4:3100
version: 1
editable: false
isDefault: false
Traefik #
Docker Compose YML #
Docker Compose file: Docker-Compose/traefik_docker-compose.tmpl
version: '3'
services:
reverse-proxy:
image: ${TRAEFIK_IMAGE}
container_name: Traefik-Reverse-Proxy
restart: unless-stopped
command:
- --api.insecure=true
- --providers.docker
- --providers.file.directory=/etc/traefik/dynamic
- --entryPoints.web.address=:80
- --entryPoints.websecure.address=:443
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- traefik
networks:
traefik:
external: true
Traefik Dockerfile #
Dockerfile for Traefik: Dockerfiles/Traefik-Dockerfile
# Use the latest Traefik image as base
FROM traefik:v2.10
# Expose ports 80,443,8080
EXPOSE 80
EXPOSE 443
EXPOSE 8080
# Copy the tls-configuration configuration file
COPY ./tls-configuration.yaml /etc/traefik/dynamic/tls-configuration.yaml
# Copy the TLS certificates
COPY ./Traefik-Certs/ /etc/certs/
# Specify the command to run on container start
CMD ["traefik", "--configFile=/etc/traefik/traefik.yml"]
Traefik Dynamic Configuration #
Traefik tls-configuration YML: tls-configuration.yaml
tls:
certificates:
- certFile: /etc/certs/fullchain.pem
keyFile: /etc/certs/privkey.pem
Git PowerShell Commands #
Install Git with Winget #
- Open PowerShell as Administrator
# Install Git
winget install Git.Git
- Close and open a new PowerShell
# Check Git version
git --version
Prerequisites #
Set the username and email address that will be associated with any commits you create across all Git repositories on your computer.
# Define your name
git config --global user.name "Your Name"
# Define youremail
git config --global user.email "you@example.com"
Repository #
Clone the repository #
# Change directory: Open a directory where the GitLab repository should be saved
cd my-git-projects
# Clone the repository
git clone git@gitlab.jklug.work:root/loki-grafana-docker-compose-stack.git
# CD into the repository
cd .\loki-grafana-docker-compose-stack\
Create / Switch Branch #
# List available branches: Local
git branch
# List available branches: Remote
git branch -r
# List available branches: Local & remote
git branch -a
# Create main branch & switch into main branch (The main branch should already be existing)
git switch --create main
# Switch to main branch
git switch main
Commit File or Folder #
# Stages the "foldername" folder and its contents for the next commit
git add .\foldername\
# Stage all new and modified files in current folder
git add .
# Commit the changes
git commit -m "Added folder"
# Push the commit to the current branch of the remote repository on GitLab
git push
# Push the commit to the main branch of the remote repository on GitLab
git push origin main
Fetch & Pull Changes #
# Verify the current branch
git branch
# Shell output:
* main
# Fetch the changes to the local branch
git fetch
# Fetch the changes to the local branch: All branches
git fetch --all
Git Fetch
Download but not merge the changes from the remote repository.
# Pull the changes to the local branch
git pull
Git Pull
Git first fetches the updates from the remote repository (the same as git fetch) and then automatically tries to merge the latest version of the branch from the remote into the current local branch.
Checkout Branch #
# Checkout specific remote branch "loki-grafana-main" to the local repository
git checkout -b loki-grafana-main origin/loki-grafana-main
Links #
# Loki & Grafana stack on GitHub
https://github.com/jueklu/loki-grafana-stack
# Loki, Grafana & Traefik stack on GitHub
https://github.com/jueklu/loki-grafana-traefik-stack