↓Skip to main content

Loki & Grafana - Kubernetes K8s Deployment with Several Loki Instances for Seperated Data Sources; Nginx Ingress with TLS Secret; Promtail Helm and Docker Container Deployment

2639 words·
Loki Grafana Promtail Kubernetes K8s Helm
Table of Contents
Loki-Stack - This article is part of a series.
Part 1: Loki & Grafana - Docker Compose Stack: Promtail Container for Container-, /var/log- and Systemd-Journal Logs; Promtail Bare-Metal for Systemd-Journal Logs; Traefik Reverse Proxy
Part 2: Loki, Grafana & Traefik - Docker Compose Stack: Deployment with GitLab CI Pipeline, Git Commands
Part 3: Loki, Grafana & Promtail Stack - Deployment with Helm Chart in a K3s Kubernetes Cluster, NodePort Service and Traefik Ingress with TLS Secret; Promtail Helm Deployment for External Loki Data Source
Part 4: This Article
Part 5: Loki: Automated Promtail Container Deployment with Ansible

Overview
#

This deployment tutorial primarily focuses on setting up a Kubernetes-based Loki & Grafana stack, featuring multiple Loki instances that serve as separate data sources for Grafana.

For a wider range of Promtail scrapping configurations check out my first post in this series: Loki & Grafana - Docker Compose Stack

In this tutorial I’m using a Kubernetes K8s cluster with MetalLB deployed with Kubespray on Debian 12 servers:

192.168.30.71 deb-02 # Controller / Master Node
192.168.30.72 deb-03 # Controller / Master Node
192.168.30.73 deb-04 # Worker Node
192.168.30.74 deb-05 # Worker Node

192.168.30.60 # NFS Server, Promtail Container for bare-metal systemd-journal scrapping
192.168.30.61 # Promtail Container for Docker container scrapping

NFS Prerequisites
#

The following steps are only necessary for the Loki deployment variant with a NFS volume for peristent data.

NFS Server Setup
#

Loki Folder Structure
#

Create the folder structure for the Loki volume mounts:

# Create folder structure for the Loki volume mapping
sudo mkdir -p /srv/nfs/k8s_share/{loki-1,loki-2,loki-3}

# Change the owner
sudo chown 10001:10001 /srv/nfs/k8s_share/{loki-1,loki-2,loki-3}

NFS Exports
#

I’m using the following NFS server configuration:

# Install NFS package
sudo apt install nfs-kernel-server

# Open NFS configuration
sudo vi /etc/exports

# NFS configuration: Define the kubernetes nodes
/srv/nfs/k8s_share 192.168.30.71(rw,sync,no_root_squash)
/srv/nfs/k8s_share 192.168.30.72(rw,sync,no_root_squash)
/srv/nfs/k8s_share 192.168.30.73(rw,sync,no_root_squash)
/srv/nfs/k8s_share 192.168.30.74(rw,sync,no_root_squash)

# Restart NFS server
sudo systemctl restart nfs-server

Install NFS on Kubernetes Nodes
#

Install the NFS utilities package on all Kubernetes nodes:

# Install NFS utilities package
sudo apt install nfs-common -y

Optional: Verify the NFS connectivity

# Verify that the NFS server is correctly configured
/usr/sbin/showmount -e 192.168.30.60

# Shell output:
Export list for 192.168.30.60:
/srv/nfs/k8s_share 192.168.30.74,192.168.30.73,192.168.30.72,192.168.30.71

Loki Grafana Stack
#

Create Namespace
# 

# Create a namespace for the Loki & Grafana deployments
kubectl create namespace loki-grafana-stack

Grafana Deployment
#

Deployment & NodePort Service
# 

# Create a configuration for the Grafana Deployment
vi grafana-deployment.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
      - name: Loki-K8s-Cluser
        type: loki
        access: proxy
        url: http://loki-1:3101
        version: 1
        editable: false
        isDefault: true

      - name: Loki-VarLog
        type: loki
        access: proxy
        url: http://loki-2:3102
        version: 1
        editable: false
        isDefault: false

      - name: Loki-Container
        type: loki
        access: proxy
        url: http://loki-3:3103
        version: 1
        editable: false
        isDefault: false    

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        env:
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: "Admin"
        - name: GF_AUTH_DISABLE_LOGIN_FORM
          value: "true"
        volumeMounts:
        - name: datasource-config
          mountPath: /etc/grafana/provisioning/datasources
          readOnly: true
      volumes:
      - name: datasource-config
        configMap:
          name: grafana-datasources

---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 30000
  selector:
    app: grafana

Kubernetes TLS Certificate Secret
#

In this setup I’m using a Let’s Encrypt wildcard certificate.

# Create a Kubernetes secret for the TLS certificate
kubectl create secret tls grafana-tls --cert=./fullchain.pem --key=./privkey.pem -n loki-grafana-stack

Nginx Ingress for NodePort Service
# 

# Create YML file
vi grafana-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  tls:
  - hosts:
    - grafana.jklug.work
    secretName: grafana-tls
  rules:
  - host: grafana.jklug.work
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: grafana
            port:
              number: 3000

DNS Entry
#

The DNS entry for the Nginx ingress must point to one of the worker nodes:

192.168.30.73 grafana.jklug.work
# Or 
192.168.30.74 grafana.jklug.work

Loki Instance 1
# 

# Create a configuration for the frist Loki instance
vi loki-1-deployment.yaml
Deployment with NodePort Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-1
  template:
    metadata:
      labels:
        app: loki-1
    spec:
      containers:
      - name: loki-1
        image: grafana/loki:latest
        ports:
        - containerPort: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki-1
spec:
  type: NodePort
  ports:
    - port: 3101
      targetPort: 3100
      nodePort: 31001
  selector:
    app: loki-1
Deployment with LoadBalancer Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-1
  template:
    metadata:
      labels:
        app: loki-1
    spec:
      containers:
      - name: loki-1
        image: grafana/loki:latest
        ports:
        - containerPort: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki-1
spec:
  type: LoadBalancer
  ports:
    - port: 3101
      targetPort: 3100
  selector:
    app: loki-1
Deployment with LoadBalancer Service & NFS Volume 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-1
  template:
    metadata:
      labels:
        app: loki-1
    spec:
      containers:
      - name: loki-1
        image: grafana/loki:latest
        ports:
        - containerPort: 3100
        volumeMounts:
          - name: nfs
            mountPath: "/loki"  # Loki data directory
      volumes:
        - name: nfs
          nfs:
            server: "192.168.30.60"
            path: "/srv/nfs/k8s_share/loki-1"

---
apiVersion: v1
kind: Service
metadata:
  name: loki-1
spec:
  type: LoadBalancer
  ports:
    - port: 3101
      targetPort: 3100
  selector:
    app: loki-1

Loki Instance 2
# 

# Create a configuration for the second Loki instance
vi loki-2-deployment.yaml
Deployment with NodePort Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-2
  template:
    metadata:
      labels:
        app: loki-2
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki-2
spec:
  type: NodePort
  ports:
    - port: 3102
      targetPort: 3100
      nodePort: 31002
  selector:
    app: loki-2
Deployment with LoadBalancer Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-2
  template:
    metadata:
      labels:
        app: loki-2
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki-2
spec:
  type: LoadBalancer
  ports:
    - port: 3102
      targetPort: 3100
  selector:
    app: loki-2
Deployment with LoadBalancer Service & NFS Volume 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-2
  template:
    metadata:
      labels:
        app: loki-2
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100
        volumeMounts:
          - name: nfs
            mountPath: "/loki"  # Loki data directory
      volumes:
        - name: nfs
          nfs:
            server: "192.168.30.60"
            path: "/srv/nfs/k8s_share/loki-2"

---
apiVersion: v1
kind: Service
metadata:
  name: loki-2
spec:
  type: LoadBalancer
  ports:
    - port: 3102
      targetPort: 3100
  selector:
    app: loki-2

Loki Instance 3
# 

# Create a configuration for the thrid Loki instance
vi loki-3-deployment.yaml
Deployment with NodePort Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-3
  template:
    metadata:
      labels:
        app: loki-3
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki-3
spec:
  type: NodePort
  ports:
    - port: 3103
      targetPort: 3100
      nodePort: 31003
  selector:
    app: loki-3
Deployment with LoadBalancer Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-3
  template:
    metadata:
      labels:
        app: loki-3
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki-3
spec:
  type: LoadBalancer
  ports:
    - port: 3103
      targetPort: 3100
  selector:
    app: loki-3
Deployment with LoadBalancer Service & NFS Volume 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki-3
  template:
    metadata:
      labels:
        app: loki-3
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100
        volumeMounts:
          - name: nfs
            mountPath: "/loki"  # Loki data directory
      volumes:
        - name: nfs
          nfs:
            server: "192.168.30.60"
            path: "/srv/nfs/k8s_share/loki-3"

---
apiVersion: v1
kind: Service
metadata:
  name: loki-3
spec:
  type: LoadBalancer
  ports:
    - port: 3103
      targetPort: 3100
  selector:
    app: loki-3

Deploy the Resources
#

Deploy the resources:

# Deploy Grafana
kubectl apply -f grafana-deployment.yaml -n loki-grafana-stack

# Deploy Grafana Nginx Ingress
kubectl apply -f grafana-ingress.yml -n loki-grafana-stack

# Deploy Loki instance 1
kubectl apply -f loki-1-deployment.yaml -n loki-grafana-stack

# Deploy Loki instance 2
kubectl apply -f loki-2-deployment.yaml -n loki-grafana-stack

# Deploy Loki instance 3
kubectl apply -f loki-3-deployment.yaml -n loki-grafana-stack

Verify the Deployment
#

Grafana Deployment, Loki Deployment with NodePort Service 
# List all resources
kubectl get all -n loki-grafana-stack

# Shell output:
NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-7fddfb7557-fqp99   1/1     Running   0          56m
pod/loki-1-5cff777949-hcstd    1/1     Running   0          112s
pod/loki-2-59f556b498-9qs2b    1/1     Running   0          106s
pod/loki-3-769b7589fc-b297d    1/1     Running   0          102s

NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/grafana   NodePort   10.233.51.156   <none>        3000:30000/TCP   56m
service/loki-1    NodePort   10.233.1.30     <none>        3101:31001/TCP   112s
service/loki-2    NodePort   10.233.48.229   <none>        3102:31002/TCP   106s
service/loki-3    NodePort   10.233.49.146   <none>        3103:31003/TCP   6s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           56m
deployment.apps/loki-1    1/1     1            1           112s
deployment.apps/loki-2    1/1     1            1           106s
deployment.apps/loki-3    1/1     1            1           102s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-7fddfb7557   1         1         1       56m
replicaset.apps/loki-1-5cff777949    1         1         1       112s
replicaset.apps/loki-2-59f556b498    1         1         1       106s
replicaset.apps/loki-3-769b7589fc    1         1         1       102s
Grafana Deployment, Loki Deployment with LoadBalancer Service 
# List all resources
kubectl get all -n loki-grafana-stack

# Shell output:
NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-7fddfb7557-fqp99   1/1     Running   0          60m
pod/loki-1-5cff777949-xwmbf    1/1     Running   0          61s
pod/loki-2-59f556b498-fjh7l    1/1     Running   0          55s
pod/loki-3-769b7589fc-6mnt9    1/1     Running   0          49s

NAME              TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
service/grafana   NodePort       10.233.51.156   <none>           3000:30000/TCP   60m
service/loki-1    LoadBalancer   10.233.49.223   192.168.30.241   3101:30896/TCP   61s
service/loki-2    LoadBalancer   10.233.56.37    192.168.30.242   3102:30187/TCP   55s
service/loki-3    LoadBalancer   10.233.59.215   192.168.30.243   3103:32203/TCP   49s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           60m
deployment.apps/loki-1    1/1     1            1           61s
deployment.apps/loki-2    1/1     1            1           55s
deployment.apps/loki-3    1/1     1            1           49s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-7fddfb7557   1         1         1       60m
replicaset.apps/loki-1-5cff777949    1         1         1       61s
replicaset.apps/loki-2-59f556b498    1         1         1       55s
replicaset.apps/loki-3-769b7589fc    1         1         1       49s
Nginx Ingress & TLS Secret 
# List the ingress resources
kubectl get ingress -n loki-grafana-stack

# Shell output:
NAME      CLASS    HOSTS                ADDRESS                       PORTS     AGE
grafana   <none>   grafana.jklug.work   192.168.30.73,192.168.30.74   80, 443   6m24s

# Verify the TLS secret
kubectl get secrets -n loki-grafana-stack

# Shell output:
NAME          TYPE                DATA   AGE
grafana-tls   kubernetes.io/tls   2      8s

Access Grafana
# 

# Access Grafana: NodePort Service
http://192.168.30.71:30000
http://192.168.30.72:30000
http://192.168.30.73:30000
http://192.168.30.74:30000

# Access Grafana: Via Nginx Ingress
https://grafana.jklug.work

Note: When accessing Grafana via the Nginx ingress, the DNS entry must point to one of the worker nodes http://192.168.30.73 or http://192.168.30.74.


Delete the Deployment
# 

# Delete the Grafana deployment
kubectl delete  -f grafana-deployment.yaml -n loki-grafana-stack

# Delete the Grafana Nginx ingress
kubectl delete ingress grafana  -n loki-grafana-stack

# Delete the TLS certificate secret
kubectl delete secret grafana-tls -n loki-grafana-stack

# Delete the Loki instance 1 deployment
kubectl delete  -f loki-1-deployment.yaml -n loki-grafana-stack

# Delete the Loki instance 2 deployment
kubectl delete  -f loki-2-deployment.yaml -n loki-grafana-stack

# Delete the Loki instance 3 deployment
kubectl delete  -f loki-3-deployment.yaml -n loki-grafana-stack



Promtail Helm Deployment for K8s Scrapping
#

This Helm chart automatically deploys the Promtail pod on all Kubernetes controller & worker nodes.

The kubernetes logs can be filtered according to the following labels: app, component, container, filename, instance, job, namespace, node_name, pod, stream

Add Grafana Repository
# 

# Add Grafana repository
helm repo add grafana https://grafana.github.io/helm-charts

# Update package index
helm repo update

List Helm Charts
# 

# List the available Loki Charts in the Grafana Repository
helm search repo loki

# Shell output:
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
grafana/loki                    6.6.2           3.0.0           Helm chart for Grafana Loki and Grafana Enterpr...
grafana/loki-canary             0.14.0          2.9.1           Helm chart for Grafana Loki Canary
grafana/loki-distributed        0.79.0          2.9.6           Helm chart for Grafana Loki in microservices mode
grafana/loki-simple-scalable    1.8.11          2.6.1           Helm chart for Grafana Loki in simple, scalable...
grafana/loki-stack              2.10.2          v2.9.3          Loki: like Prometheus, but for logs.
grafana/fluent-bit              2.6.0           v2.1.0          Uses fluent-bit Loki go plugin for gathering lo...
grafana/lgtm-distributed        1.0.1           6.59.4          Umbrella chart for a distributed Loki, Grafana,...
grafana/meta-monitoring         1.0.0           0.0.1           A Helm chart for meta monitoring Grafana Loki, ...
grafana/promtail                6.15.5          2.9.3           Promtail is an agent which ships the contents o..

Save Promtail Values
# 

# Save the Promtail deployment configuration
helm show values grafana/promtail > promtail-values.yaml

Adopt Promtail Values
#

Adopt the Promtail clients URL to point to the Loki instance:

# Edit the Promtail deployment configuration
vi promtail-values.yaml
Loki with NodePort Service 
# Define the IP and port of the Loki instance
  clients:
    - url: http://192.168.30.71:31001/loki/api/v1/push
Loki with LoadBalancer Service 
# Define the IP and port of the Loki instance
  clients:
    - url: http://192.168.30.241:3101/loki/api/v1/push

Deploy Promtail
# 

# Deploy Promtail
helm install --values promtail-values.yaml promtail grafana/promtail
Shell Output 
# Shell output:
NAME: promtail
LAST DEPLOYED: Mon Jun  3 18:10:48 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
***********************************************************************
 Welcome to Grafana Promtail
 Chart version: 6.15.5
 Promtail version: 2.9.3
***********************************************************************

Verify the application is working by running these commands:
* kubectl --namespace default port-forward daemonset/promtail 3101
* curl http://127.0.0.1:3101/metrics

Delete Promtail Deployment
# 

# Delete the Promtail deployment
helm uninstall promtail



Promtail Container for Bare-Metal Scrapping
#

Prerequisites
#

Docker System User
# 

# Create a system user for Docker
sudo adduser --system --ingroup docker --shell /sbin/nologin docker-system

# Add the user to the systemd-journal & adm group
sudo usermod -aG systemd-journal docker-system

# Reboot the system to apply the group changes
sudo reboot

# Verify the user groups
groups docker-system

# Shell output
docker-system : docker systemd-journal

Promtail Container
#

Folder Structure
# 

# Create folder struture
sudo mkdir -p /opt/promtail && cd /opt/promtail

Promtail Configuration
# 

# Create Promtail configuration file
sudo vi promtail.yaml
Promtail Config: Loki with NodePort Service 
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://192.168.30.71:31002/loki/api/v1/push

scrape_configs:
  - job_name: systemd-journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
      path: /var/log/journal
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: "unit"
      - source_labels: ["__journal__hostname"]
        target_label: host
      - source_labels: ["__journal_priority_keyword"]
        target_label: level
      - source_labels: ["__journal_syslog_identifier"]
        target_label: syslog_identifier
Promtail Config: Loki with LoadBalancer Service 
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://192.168.30.242:3102/loki/api/v1/push

scrape_configs:
  - job_name: systemd-journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
      path: /var/log/journal
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: "unit"
      - source_labels: ["__journal__hostname"]
        target_label: host
      - source_labels: ["__journal_priority_keyword"]
        target_label: level
      - source_labels: ["__journal_syslog_identifier"]
        target_label: syslog_identifier

Docker Compose File
# 

# Create Docker Compose file
sudo vi docker-compose.yml

services:
  promtail:
    image:  grafana/promtail:latest
    container_name: promtail
    volumes:
      - ./promtail.yaml:/etc/promtail/docker-config.yaml
      - /var/log/journal:/var/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
    command: -config.file=/etc/promtail/docker-config.yaml
    restart: unless-stopped
    networks:
      - loki

networks:
  loki:
    name: loki

Change Ownership
# 

# Change ownership
sudo chown -R docker-system:docker /opt/promtail

Create & Start Docker Stack
# 

# Create & start containers
sudo -u docker-system docker compose up -d



Promtail Container for Containers
#

Prerequisites
#

Docker Network
# 

# Create network used for Promtail to communicate with other Docker containers
sudo docker network create promtail

Docker System User
# 

# Create a system user for Docker
sudo adduser --system --ingroup docker --shell /sbin/nologin docker-system

Promtail Docker Compose
#

Folder Structure
# 

# Create folder struture
sudo mkdir -p /opt/promtail && cd /opt/promtail

Promtail Configuration
# 

# Create the Promtail configuration file
sudo vi promtail.yaml
Loki with NodePort Service 
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://192.168.30.71:31003/loki/api/v1/push

scrape_configs:
  - job_name: docker_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'
Loki with LoadBalancer Service 
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://192.168.30.243:3103/loki/api/v1/push

scrape_configs:
  - job_name: docker_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'
  • Docker socket is mounted to Promtail so that it’s aware of all the docker events. It’s configured to filter only containers with docker label logging=promtail.

Docker Compose File
# 

# Create Docker Compose file
sudo vi docker-compose.yml

services:
  promtail:
    image:  grafana/promtail:latest
    container_name: promtail
    volumes:
      - ./promtail.yaml:/etc/promtail/docker-config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/docker-config.yaml
    restart: unless-stopped
    networks:
      - promtail

networks:
  promtail:
    external: true

Change Ownership
# 

# Change ownership
sudo chown -R docker-system:docker /opt/promtail

Create & Start Docker Stack
# 

# Create & start containers
sudo -u docker-system docker compose up -d

Example Container
#

Folder Structure
# 

# Create folder struture
sudo mkdir -p /opt/some-containers && cd /opt/some-containers

Docker Compose File
# 

# Create Docker Compose file
sudo vi docker-compose.yml

services:
  nginx:
    container_name: container-1
    image: nginx:latest
    labels:
      logging: "promtail"
      logging_jobname: "containerlogs"
    ports:
      - 8080:80
    command: /bin/sh -c "echo 'Web-App-1 Nginx Container' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
    restart: unless-stopped
    networks:
      - intern
      - promtail

  apache2:
    container_name: container-2
    image: httpd:latest
    labels:
      logging: "promtail"
      logging_jobname: "containerlogs"
    ports:
      - "8081:80"
    command: >
            bash -c "echo 'Web-App-2 Apache2 Container' > /usr/local/apache2/htdocs/index.html && httpd-foreground"
    restart: unless-stopped
    networks:
      - intern
      - promtail

networks:
  promtail:
    external: true
  intern:

Change Ownership
# 

# Change ownership
sudo chown -R docker-system:docker /opt/some-containers

Create & Start Docker Stack
# 

# Create & start containers
sudo -u docker-system docker compose up -d
Loki-Stack - This article is part of a series.
Part 1: Loki & Grafana - Docker Compose Stack: Promtail Container for Container-, /var/log- and Systemd-Journal Logs; Promtail Bare-Metal for Systemd-Journal Logs; Traefik Reverse Proxy
Part 2: Loki, Grafana & Traefik - Docker Compose Stack: Deployment with GitLab CI Pipeline, Git Commands
Part 3: Loki, Grafana & Promtail Stack - Deployment with Helm Chart in a K3s Kubernetes Cluster, NodePort Service and Traefik Ingress with TLS Secret; Promtail Helm Deployment for External Loki Data Source
Part 4: This Article
Part 5: Loki: Automated Promtail Container Deployment with Ansible