Overview #
Prometheus and Node Exporter are deployed via Helm chart, for Grafana I’m using a dedicated deployment configuration. This approach allows for an easy adaptation of Grafana’s data source configuration and enables its integration with other data sources, such as Loki.
For a more comprehensive description regarding the Grafana Kubernetes deployment check out my blog post titled “Loki & Grafana - Kubernetes K8s Deployment”.
In this tutorial I’m using a Kubernetes K8s cluster with MetalLB deployed with Kubespray on Debian 12 servers:
192.168.30.71 deb-02 # Controller / Master Node
192.168.30.72 deb-03 # Controller / Master Node
192.168.30.73 deb-04 # Worker Node
192.168.30.74 deb-05 # Worker Node
Prerequisites #
Create Namespace #
# Create a namespace for the Prometheus & Grafana deployments
kubectl create namespace prometheus-grafana-stack
Grafana Deployment #
Deployment & NodePort Service #
# Create a configuration for the Grafana Deployment
vi grafana-deployment.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
data:
datasources.yaml: |
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://my-prometheus-kube-prometh-prometheus:9090
isDefault: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:latest
ports:
- containerPort: 3000
env:
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
value: "Admin"
- name: GF_AUTH_DISABLE_LOGIN_FORM
value: "true"
volumeMounts:
- name: datasource-config
mountPath: /etc/grafana/provisioning/datasources
readOnly: true
volumes:
- name: datasource-config
configMap:
name: grafana-datasources
---
apiVersion: v1
kind: Service
metadata:
name: grafana
spec:
type: NodePort
ports:
- port: 3000
targetPort: 3000
nodePort: 30000
selector:
app: grafana
Deploy the Grafana Resources #
# Deploy Grafana
kubectl apply -f grafana-deployment.yaml -n prometheus-grafana-stack
Access Grafana #
# Access Grafana: NodePort Service
http://192.168.30.71:30000
http://192.168.30.72:30000
http://192.168.30.73:30000
http://192.168.30.74:30000
Prometheus & Node Exporter Helm Deployments #
Add Prometheus Repository #
# Add the Prometheus Community repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# Update package index
helm repo update
List Helm Charts #
# List the available Prometheus Charts
helm search repo prometheus
# Shell output:
NAME CHART VERSION APP VERSION DESCRIPTION
prometheus-community/kube-prometheus-stack 59.1.0 v0.74.0 kube-prometheus-stack collects Kubernetes manif...
prometheus-community/prometheus 25.21.0 v2.52.0 Prometheus is a monitoring system and time seri...
prometheus-community/prometheus-adapter 4.10.0 v0.11.2 A Helm chart for k8s prometheus adapter
prometheus-community/prometheus-blackbox-exporter 8.17.0 v0.25.0 Prometheus Blackbox Exporter
prometheus-community/prometheus-cloudwatch-expo... 0.25.3 0.15.5 A Helm chart for prometheus cloudwatch-exporter
prometheus-community/prometheus-conntrack-stats... 0.5.10 v0.4.18 A Helm chart for conntrack-stats-exporter
prometheus-community/prometheus-consul-exporter 1.0.0 0.4.0 A Helm chart for the Prometheus Consul Exporter
prometheus-community/prometheus-couchdb-exporter 1.0.0 1.0 A Helm chart to export the metrics from couchdb...
prometheus-community/prometheus-druid-exporter 1.1.0 v0.11.0 Druid exporter to monitor druid metrics with Pr...
prometheus-community/prometheus-elasticsearch-e... 5.8.0 v1.7.0 Elasticsearch stats exporter for Prometheus
prometheus-community/prometheus-fastly-exporter 0.3.0 v7.6.1 A Helm chart for the Prometheus Fastly Exporter
prometheus-community/prometheus-ipmi-exporter 0.3.0 v1.8.0 This is an IPMI exporter for Prometheus.
prometheus-community/prometheus-json-exporter 0.11.0 v0.6.0 Install prometheus-json-exporter
prometheus-community/prometheus-kafka-exporter 2.10.0 v1.7.0 A Helm chart to export the metrics from Kafka i...
prometheus-community/prometheus-memcached-exporter 0.3.2 v0.14.3 Prometheus exporter for Memcached metrics
prometheus-community/prometheus-modbus-exporter 0.1.2 0.4.1 A Helm chart for prometheus-modbus-exporter
prometheus-community/prometheus-mongodb-exporter 3.5.0 0.40.0 A Prometheus exporter for MongoDB metrics
prometheus-community/prometheus-mysql-exporter 2.5.3 v0.15.1 A Helm chart for prometheus mysql exporter with...
prometheus-community/prometheus-nats-exporter 2.17.0 0.15.0 A Helm chart for prometheus-nats-exporter
prometheus-community/prometheus-nginx-exporter 0.2.1 0.11.0 A Helm chart for the Prometheus NGINX Exporter
prometheus-community/prometheus-node-exporter 4.34.0 1.8.0 A Helm chart for prometheus node-exporter
prometheus-community/prometheus-opencost-exporter 0.1.1 1.108.0 Prometheus OpenCost Exporter
prometheus-community/prometheus-operator 9.3.2 0.38.1 DEPRECATED - This chart will be renamed. See ht...
prometheus-community/prometheus-operator-admiss... 0.12.0 0.73.0 Prometheus Operator Admission Webhook
prometheus-community/prometheus-operator-crds 12.0.0 v0.74.0 A Helm chart that collects custom resource defi...
prometheus-community/prometheus-pgbouncer-exporter 0.3.0 v0.8.0 A Helm chart for prometheus pgbouncer-exporter
prometheus-community/prometheus-pingdom-exporter 2.5.0 20190610-1 A Helm chart for Prometheus Pingdom Exporter
prometheus-community/prometheus-pingmesh-exporter 0.4.0 v1.2.1 Prometheus Pingmesh Exporter
prometheus-community/prometheus-postgres-exporter 6.0.0 v0.15.0 A Helm chart for prometheus postgres-exporter
prometheus-community/prometheus-pushgateway 2.12.0 v1.8.0 A Helm chart for prometheus pushgateway
prometheus-community/prometheus-rabbitmq-exporter 1.11.0 v0.29.0 Rabbitmq metrics exporter for prometheus
prometheus-community/prometheus-redis-exporter 6.2.0 v1.58.0 Prometheus exporter for Redis metrics
prometheus-community/prometheus-smartctl-exporter 0.9.0 v0.12.0 A Helm chart for Kubernetes
prometheus-community/prometheus-snmp-exporter 5.4.0 v0.26.0 Prometheus SNMP Exporter
prometheus-community/prometheus-sql-exporter 0.1.0 v0.5.4 Prometheus SQL Exporter
prometheus-community/prometheus-stackdriver-exp... 4.5.0 v0.15.0 Stackdriver exporter for Prometheus
prometheus-community/prometheus-statsd-exporter 0.13.1 v0.26.1 A Helm chart for prometheus stats-exporter
prometheus-community/prometheus-systemd-exporter 0.2.2 0.6.0 A Helm chart for prometheus systemd-exporter
prometheus-community/prometheus-to-sd 0.4.2 0.5.2 Scrape metrics stored in prometheus format and ...
prometheus-community/prometheus-windows-exporter 0.3.1 0.25.1 A Helm chart for prometheus windows-exporter
prometheus-community/alertmanager 1.11.0 v0.27.0 The Alertmanager handles alerts sent by client ...
prometheus-community/alertmanager-snmp-notifier 0.3.0 v1.5.0 The SNMP Notifier handles alerts coming from Pr...
prometheus-community/jiralert 1.7.1 v1.3.0 A Helm chart for Kubernetes to install jiralert
prometheus-community/kube-state-metrics 5.19.1 2.12.0 Install kube-state-metrics to generate and expo...
prometheus-community/prom-label-proxy 0.7.2 v0.8.1 A proxy that enforces a given label in a given ...
Save Prometheus-Stack Values #
# Save the Prometheus deployment configuration
helm show values prometheus-community/kube-prometheus-stack > prometheus-values.yaml
Adopt Prometheus-Stack Values #
# Edit the Prometheus deployment configuration
vi prometheus-values.yaml
# Disable Grafana:
grafana:
enabled: false
Deploy Prometheus #
# Deploy Prometheus: With adopted values
helm install --values prometheus-values.yaml my-prometheus prometheus-community/kube-prometheus-stack -n prometheus-grafana-stack
Shell Output
# Shell output:
configmap/grafana-datasources created
deployment.apps/grafana created
service/grafana created
debian@node1:~/loki-grafana-kubernetes$ helm install --values prometheus-values.yaml my-prometheus prometheus-community/kube-prometheus-stack -n prometheus-grafana-stack
NAME: my-prometheus
LAST DEPLOYED: Tue Jun 4 20:14:55 2024
NAMESPACE: prometheus-grafana-stack
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace prometheus-grafana-stack get pods -l "release=my-prometheus"
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
Verify the Deployment #
# Verify Resources
kubectl get all -n prometheus-grafana-stack
# Shell output:
NAME READY STATUS RESTARTS AGE
pod/alertmanager-my-prometheus-kube-prometh-alertmanager-0 2/2 Running 0 55s
pod/grafana-7fddfb7557-8dknn 1/1 Running 0 82s
pod/my-prometheus-kube-prometh-operator-7cc54fff54-xsvln 1/1 Running 0 57s
pod/my-prometheus-kube-state-metrics-64d9b6dfc5-zpjtx 1/1 Running 0 57s
pod/my-prometheus-prometheus-node-exporter-6lwht 1/1 Running 0 57s
pod/my-prometheus-prometheus-node-exporter-fc2xv 1/1 Running 0 56s
pod/my-prometheus-prometheus-node-exporter-h2lvl 1/1 Running 0 57s
pod/my-prometheus-prometheus-node-exporter-hwj98 1/1 Running 0 57s
pod/prometheus-my-prometheus-kube-prometh-prometheus-0 2/2 Running 0 55s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 55s
service/grafana NodePort 10.233.12.137 <none> 3000:30000/TCP 82s
service/my-prometheus-kube-prometh-alertmanager ClusterIP 10.233.47.170 <none> 9093/TCP,8080/TCP 57s
service/my-prometheus-kube-prometh-operator ClusterIP 10.233.60.244 <none> 443/TCP 57s
service/my-prometheus-kube-prometh-prometheus ClusterIP 10.233.11.220 <none> 9090/TCP,8080/TCP 57s
service/my-prometheus-kube-state-metrics ClusterIP 10.233.56.56 <none> 8080/TCP 57s
service/my-prometheus-prometheus-node-exporter ClusterIP 10.233.34.40 <none> 9100/TCP 57s
service/prometheus-operated ClusterIP None <none> 9090/TCP 55s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/my-prometheus-prometheus-node-exporter 4 4 4 4 4 kubernetes.io/os=linux 57s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 82s
deployment.apps/my-prometheus-kube-prometh-operator 1/1 1 1 57s
deployment.apps/my-prometheus-kube-state-metrics 1/1 1 1 57s
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-7fddfb7557 1 1 1 82s
replicaset.apps/my-prometheus-kube-prometh-operator-7cc54fff54 1 1 1 57s
replicaset.apps/my-prometheus-kube-state-metrics-64d9b6dfc5 1 1 1 57s
NAME READY AGE
statefulset.apps/alertmanager-my-prometheus-kube-prometh-alertmanager 1/1 55s
statefulset.apps/prometheus-my-prometheus-kube-prometh-prometheus 1/1 55s
Delete the Deployment #
# Delete the Grafana deployment
kubectl delete -f grafana-deployment.yaml -n prometheus-grafana-stack
# Delete the Prometheus Deployment
helm uninstall my-prometheus -n prometheus-grafana-stack
Grafana Dashboards #
Import Dashboard #
-
Go to: “Dashboard” > “New” > “Import”
-
Grafana.com dashboard URL or ID: (Paste the dashboard ID)
-
Click “Load”
-
Seletct the Data Source “Prometheus”
-
Click “Import”
Example Dashboards #
Node Exporter Full #
Detailed visualizations of the metrics collected via the Prometheus Node Exporter across the different Kubernetes nodes.
Dashboard ID: 1860
