↓Skip to main content

Loki, Grafana & Promtail Stack - Deployment with Helm Chart in a K3s Kubernetes Cluster, NodePort Service and Traefik Ingress with TLS Secret; Promtail Helm Deployment for External Loki Data Source

1173 words·
Loki Grafana Promtail Helm Kubernetes K3s
Table of Contents
Loki-Stack - This article is part of a series.
Part 1: Loki & Grafana - Docker Compose Stack: Promtail Container for Container-, /var/log- and Systemd-Journal Logs; Promtail Bare-Metal for Systemd-Journal Logs; Traefik Reverse Proxy
Part 2: Loki, Grafana & Traefik - Docker Compose Stack: Deployment with GitLab CI Pipeline, Git Commands
Part 3: This Article
Part 4: Loki & Grafana - Kubernetes K8s Deployment with Several Loki Instances for Seperated Data Sources; Nginx Ingress with TLS Secret; Promtail Helm and Docker Container Deployment
Part 5: Loki: Automated Promtail Container Deployment with Ansible

This is a very useful way to deploy Promtail, as each Kubernetes node automatically get’s a Promtail deployment, that scrapts it’s logs.

Kubernetes Cluster
#

I’m using a 3 node Kubernetes cluster based on K3s installed on Debian 12 servers:

debian-01 192.168.30.20
debian-02 192.168.30.21
debian-03 192.168.30.22

# List Kubernetes nodes
k3s kubectl get nodes

# Shell output:
NAME        STATUS   ROLES                  AGE     VERSION
debian-01   Ready    control-plane,master   2m34s   v1.29.4+k3s1
debian-02   Ready    worker                 53s     v1.29.4+k3s1
debian-03   Ready    worker                 49s     v1.29.4+k3s1

Helm
#

Install Helm
# 

# Install Helm with script
cd /tmp && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&
chmod +x get_helm.sh &&
./get_helm.sh

# Verify the installation / check version
helm version

Kubectl Environment Variable
# 

# Set environment variable for the kubectl configuration: Temporary
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# Set environment variable for the kubectl configuration: Permanent
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc

# Apply changes
source ~/.bashrc

Add Grafana Repository
# 

# Add Grafana repository
helm repo add grafana https://grafana.github.io/helm-charts

# Update package index
helm repo update

Deploy Loki-Stack
#

Find Loki-Stack Repository
# 

# Search for Loki repositories
helm search repo loki

# Shell output:
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
grafana/loki                    6.5.2           3.0.0           Helm chart for Grafana Loki and Grafana Enterpr...
grafana/loki-canary             0.14.0          2.9.1           Helm chart for Grafana Loki Canary
grafana/loki-distributed        0.79.0          2.9.6           Helm chart for Grafana Loki in microservices mode
grafana/loki-simple-scalable    1.8.11          2.6.1           Helm chart for Grafana Loki in simple, scalable...
grafana/loki-stack              2.10.2          v2.9.3          Loki: like Prometheus, but for logs.
grafana/fluent-bit              2.6.0           v2.1.0          Uses fluent-bit Loki go plugin for gathering lo...
grafana/lgtm-distributed        1.0.1           6.59.4          Umbrella chart for a distributed Loki, Grafana,...
grafana/meta-monitoring         1.0.0           0.0.1           A Helm chart for meta monitoring Grafana Loki, ...
grafana/promtail                6.15.5          2.9.3           Promtail is an agent which ships the contents o...

Adopt Deployment Configuration
# 

# Save the Promtail deployment configuration
helm show values grafana/loki-stack > loki-stack-values.yaml

# Edit the Loki-Stack deployment configuration
vi loki-stack-values.yaml

Change the following settings:

# Change the settings for Grafana
grafana:
  enabled: true # Set to true, to enable Grafana
  sidecar:
    datasources:
      label: ""
      labelValue: ""
      enabled: true
      maxLines: 1000
  image:
    tag: latest # Set tag to latest

Deploy Loki-Stack
# 

# Deploy the Loki-Stacl
helm install --values loki-stack-values.yaml loki-stack grafana/loki-stack

Undeploy Loki-Stack
#

Use the following command to undeploy / uninstall the Loki-Stack

# Uninstall the Loki-Stack
helm uninstall loki-stack

List Pods
#

Each node in the Kubernetes cluster gets automatically a Promtail deployment:

# List the Loki-Stack pods
kubectl get pods -o wide

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
loki-stack-grafana-58cc6cc4bf-5vfrl   2/2     Running   0          36m   10.42.1.5    debian-02   <none>           <none>
loki-stack-promtail-cg4dg             1/1     Running   0          36m   10.42.0.10   debian-01   <none>           <none>
loki-stack-promtail-mkc7q             1/1     Running   0          36m   10.42.1.6    debian-02   <none>           <none>
loki-stack-promtail-dh7b5             1/1     Running   0          36m   10.42.2.5    debian-03   <none>           <none>
loki-stack-0                          1/1     Running   0          36m   10.42.2.6    debian-03   <none>           <none>

Grafana Secret
# 

# Lists all secret objects in the current namespace
kubectl get secret

# Shell output:
NAME                               TYPE                 DATA   AGE
loki-stack-grafana                 Opaque               3      7s
loki-stack                         Opaque               1      7s
loki-stack-promtail                Opaque               1      7s
sh.helm.release.v1.loki-stack.v1   helm.sh/release.v1   1      7s

# List secret details
kubectl describe secret loki-stack-grafana 

# Shell output:
Name:         loki-stack-grafana
Namespace:    default
Labels:       app.kubernetes.io/instance=loki-stack
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=grafana
              app.kubernetes.io/version=latest
              helm.sh/chart=grafana-6.43.5
Annotations:  meta.helm.sh/release-name: loki-stack
              meta.helm.sh/release-namespace: default

Type:  Opaque

Data
====
admin-password:  40 bytes
admin-user:      5 bytes
ldap-toml:       0 bytes

# List & decode the Grafana admin password
kubectl get secret loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode

# Shell output:
q5UIEkHAGXwlX3GEUmrqb5qUqQjvYXa5p7JyJnDn

Optional: Promtail Configuration
# 

# List secret details
kubectl describe secret loki-stack-promtail

# Shell output:
Name:         promtail-loki
Namespace:    default
Labels:       app=loki
              app.kubernetes.io/managed-by=Helm
              chart=loki-2.16.0
              heritage=Helm
              release=promtail
Annotations:  meta.helm.sh/release-name: promtail
              meta.helm.sh/release-namespace: default

Type:  Opaque

Data
====
loki.yaml:  1101 bytes

# List & decode the Promtail configuration
kubectl get secret loki-stack-promtail -o jsonpath="{.data.promtail\.yaml}" | base64 --decode

Access Grafana
#

Temporary: Port Forwarding
# 

# Lists pods
kubectl get pod

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE
loki-stack-grafana-58cc6cc4bf-5vfrl   2/2     Running   0          2m24s
loki-stack-promtail-cg4dg             1/1     Running   0          2m24s
loki-stack-promtail-mkc7q             1/1     Running   0          2m24s
loki-stack-promtail-dh7b5             1/1     Running   0          2m24s
loki-stack-0

# Create a port forwarding that listens on all interfaces
kubectl port-forward --address 0.0.0.0 loki-stack-grafana-58cc6cc4bf-5vfrl 3000:3000

# Stop port forwarding
Strg + c

NodePort Service
#

The NodePort Service makes the deployment accessible from a defined port of any node within the Kubernetes cluster.

# List pod labels
kubectl get pods  --show-labels

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE   LABELS
loki-stack-grafana-58cc6cc4bf-5vfrl   2/2     Running   0          28m   app.kubernetes.io/instance=loki-stack,app.kubernetes.io/name=grafana,pod-template-hash=58cc6cc4bf
loki-stack-promtail-cg4dg             1/1     Running   0          28m   app.kubernetes.io/instance=loki-stack,app.kubernetes.io/name=promtail,controller-revision-hash=6db87d799f,pod-template-generation=1
loki-stack-promtail-mkc7q             1/1     Running   0          28m   app.kubernetes.io/instance=loki-stack,app.kubernetes.io/name=promtail,controller-revision-hash=6db87d799f,pod-template-generation=1
loki-stack-promtail-dh7b5             1/1     Running   0          28m   app.kubernetes.io/instance=loki-stack,app.kubernetes.io/name=promtail,controller-revision-hash=6db87d799f,pod-template-generation=1
loki-stack-0                          1/1     Running   0          28m   app=loki,apps.kubernetes.io/pod-index=0,controller-revision-hash=loki-stack-5df499458f,name=loki-stack,release=loki-stack,statefulset.kubernetes.io/pod-name=loki-stack-0

# Create a configuration file for the Grafana NodePort
vi grafana-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  name: grafana-nodeport
spec:
  type: NodePort
  ports:
    - port: 3000
      nodePort: 30000
      protocol: TCP
  selector:
    app.kubernetes.io/name: grafana # Define Grafana pod label

# Deploy the Grafana NodePort
kubectl apply -f grafana-nodeport.yaml

# Delete the Grafana NodePort
kubectl delete service grafana-nodeport

Optional: Ingres with TLS for NodePort
#

Since I’m deploying the Loki-Stack locally in my homelab, I’m using a Let’s Encrypt wildcard certificate to enable TLS. This is my preferred way to test deployments with TLS encryption, since it’s less of a hassle than a CA.

Hosts Entry
# 

# Create a hosts entry for Grafana
192.168.30.20 grafana.jklug.work

Kubernetes TLS Certificate Secret
# 

# Create a Kubernetes secret for the TLS certificate
kubectl create secret tls grafana-tls --cert=./fullchain.pem --key=./privkey.pem

Traefik Ingres Resource
#

Create a Traefik Ingres resource:

# Create YML file
vi grafana-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
spec:
  tls:
  - hosts:
    - grafana.jklug.work
    secretName: grafana-tls
  rules:
  - host: grafana.jklug.work
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: grafana-nodeport
            port:
              number: 3000

# Apply configuration
kubectl apply -f grafana-ingress.yml

Access Grafana
# 

# Access Grafana: Port forwarding
http://192.168.30.20:3000/login

# Access Grafana: NodePort
http://192.168.30.20:30000/login
http://192.168.30.21:30000/login
http://192.168.30.22:30000/login

# Access Grafana: NodePort Ingres
https://grafana.jklug.work/login

# Default user:
admin

# Password from Grafana secret
q5UIEkHAGXwlX3GEUmrqb5qUqQjvYXa5p7JyJnDn

Grafana automatically adds Loki as the data store.

In the “Explore” section of Grafna you should be able to query the logs of all Kubernetes nodes as follows:


Deploy Promtail Only
#

This version uses an external Loki instance, that is not hosted in the Kubernetes cluster.

# Search for Loki repositories
helm search repo loki

# Shell output:
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
grafana/loki                    6.5.2           3.0.0           Helm chart for Grafana Loki and Grafana Enterpr...
grafana/loki-canary             0.14.0          2.9.1           Helm chart for Grafana Loki Canary
grafana/loki-distributed        0.79.0          2.9.6           Helm chart for Grafana Loki in microservices mode
grafana/loki-simple-scalable    1.8.11          2.6.1           Helm chart for Grafana Loki in simple, scalable...
grafana/loki-stack              2.10.2          v2.9.3          Loki: like Prometheus, but for logs.
grafana/fluent-bit              2.6.0           v2.1.0          Uses fluent-bit Loki go plugin for gathering lo...
grafana/lgtm-distributed        1.0.1           6.59.4          Umbrella chart for a distributed Loki, Grafana,...
grafana/meta-monitoring         1.0.0           0.0.1           A Helm chart for meta monitoring Grafana Loki, ...
grafana/promtail                6.15.5          2.9.3           Promtail is an agent which ships the contents o...

# Save the Promtail deployment configuration
helm show values grafana/promtail > promtail-values.yaml

# Edit the Promtail deployment configuration
vi promtail-values.yaml

# Define the IP and Port of the external Loki server
  clients:
    - url: http://192.168.30.70:3100/loki/api/v1/push

# Deploy Promtail
helm install --values promtail-values.yaml promtail grafana/promtail
Loki-Stack - This article is part of a series.
Part 1: Loki & Grafana - Docker Compose Stack: Promtail Container for Container-, /var/log- and Systemd-Journal Logs; Promtail Bare-Metal for Systemd-Journal Logs; Traefik Reverse Proxy
Part 2: Loki, Grafana & Traefik - Docker Compose Stack: Deployment with GitLab CI Pipeline, Git Commands
Part 3: This Article
Part 4: Loki & Grafana - Kubernetes K8s Deployment with Several Loki Instances for Seperated Data Sources; Nginx Ingress with TLS Secret; Promtail Helm and Docker Container Deployment
Part 5: Loki: Automated Promtail Container Deployment with Ansible