↓Skip to main content

Prometheus - Bare Metal and Docker Compose, Prometheus Exporter, Grafana Dashboard

818 words·
Prometheus Exporter Grafana Docker-Compose Monitoring
Table of Contents

Prometheus is a great tool for monitoring container and microservices.

Prerequisites
#

I’m using three Debian 12 Bookworm servers in this tutorial. The first server is used to host Prometheus and the other two servers are used to install the Node Exporter that scraps the data from the OS.

192.168.30.60 Prometheus
192.168.30.61 Exporter Node 1
192.168.30.62 Exporter Node 2

Prometheus Bare Metal
#

Download Prometheus
# 

# Copy the download link:
https://prometheus.io/download/

# Change directory
cd /tmp

# Download the tar file
sudo wget https://github.com/prometheus/prometheus/releases/download/v2.51.0-rc.0/prometheus-2.51.0-rc.0.linux-amd64.tar.gz

# Unpack the tar file
sudo tar xvfz prometheus* -C /opt

# Change directory
cd /opt/prometheus-2.51.0-rc.0.linux-amd64

Configuration
#

The following configuration monitors Prometheus itself and and the second debian server:

# Open the configuration file and delete it's content
sudo vi prometheus.yml
  • Define the targets where Prometheus should scrape the data
# prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'Prometheus-Server'
    static_configs:
      - targets:
        -  localhost:9090
  - job_name: 'Linux-Server'
    static_configs:
      - targets:
        -  192.168.30.61:9100
        -  192.168.30.62:9100

Note: The default scrape interval is 1 minute.

Start Prometheus
# 

# Start Prometheus
sudo ./prometheus

Prometheus Docker Compose
#

Folder Structure
# 

# Create folder structure
sudo mkdir -p /opt/prometheus/{prometheus_conf,prometheus_data} && cd /opt/prometheus

# Change owner
sudo chown 65534:65534 /opt/prometheus/prometheus_data

Docker Compose
# 

# Create Docker Compose File
sudo vi docker-compose.yml

version: '3.9'
services:

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus_conf:/etc/prometheus
      - ./prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
    restart: unless-stopped

Configuration File
# 

# Create Configuration
sudo vi prometheus_conf/prometheus.yml

# prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'Prometheus-Server'
    static_configs:
      - targets:
        -  localhost:9090
  - job_name: 'Linux-Server'
    static_configs:
      - targets:
        -  192.168.30.61:9100
        -  192.168.30.62:9100

Start Docker Container
# 

# Create / start Docker container
sudo docker compose up -d

Node Exporter
#

The Node Exporter is used to monitor bare metal and virtual servers, it provides metrics such as CPU, memory, disk space, disk I/O, and network bandwidth. It does not provide metrics about individual processes or applications. Applications and services should be monitored directly and not together with the host.

Installation
# 

# Copy the download link:
https://prometheus.io/download/#node_exporter

# Change directory
cd /tmp

# Download the tar file
sudo wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

# Unpack the tar file
sudo tar xvfz node_exporter* -C /opt

# Change directory
cd /opt/node_exporter*

# Start the Exporter
sudo ./node_exporter

Testing
#

  • Test the Exporter: Shell
# Install the curl package
sudo apt install curl -y

# Check the output from the exporer
curl http://localhost:9100/metrics | grep "node_"
  • Test the Exporer: Browser
# Check the Exporter metrics from the browser
http://192.168.30.61:9100/metrics

Prometheus Webinterface
# 

# Open the webinterface
http://192.168.30.60:9090

Check Targets
#

Go to Status > Targets


Query Instance State
# 

# List up instances
up

# List down instances
up == 0

Alerts
#

Docker Compose
# 

version: '3.9'
services:

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus_conf:/etc/prometheus
      - ./prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    restart: unless-stopped

prometheus.yml
# 

# Open prometheus.yml
sudo vi prometheus_conf/rules.yml

global:
  scrape_interval: 10s
  evaluation_interval: 10s

rule_files:
  - rules.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 192.168.30.61:9093
      -  192.168.30.62:9100

scrape_configs:
  - job_name: 'Prometheus-Server'
    static_configs:
      - targets:
        -  localhost:9090
  - job_name: 'Linux-Server'
    static_configs:
      - targets:
        -  192.168.30.61:9100
        -  192.168.30.62:9100

rules.yml
# 

# Create rules.yml
sudo vi prometheus_conf/rules.yml

groups:
  - name: example
    rules:
    - alert: InstanceDown
      expr: up == 0
      for: 1m

Webinterface
#

Go to Alterts > InstanceDown


Prometheus with Grafana
#

Folder Structure
# 

# Create folder structure
sudo mkdir -p /opt/prometheus/{prometheus_conf,prometheus_data,grafana_data,grafana_provisioning/datasources} && cd /opt/prometheus

# Change owner
sudo chown 65534:65534 /opt/prometheus/prometheus_data &&
sudo chown 472:root /opt/prometheus/grafana_data

Docker Compose
# 

# Create Docker Compose File
sudo vi docker-compose.yml

version: '3.9'
services:

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus_conf:/etc/prometheus
      - ./prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - 3000:3000
    depends_on:
      - prometheus
    volumes:
      - ./grafana_data:/var/lib/grafana
      - ./grafana_provisioning:/etc/grafana/provisioning
    restart: unless-stopped

Configuration Files
# 

# Create Prometheus configuration
sudo vi prometheus_conf/prometheus.yml

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'Prometheus-Server'
    static_configs:
      - targets:
        -  localhost:9090
  - job_name: 'Linux-Server'
    static_configs:
      - targets:
        -  192.168.30.61:9100
        -  192.168.30.62:9100

# Create Grafana data source configuration
sudo vi grafana_provisioning/datasources/prometheus_ds.yml

datasources:
  - name: Prometheus
    access: proxy
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

Start Docker Container
# 

# Create / start Docker container
sudo docker compose up -d

Grafana Webinterface
# 

# Grafana webinterface
192.168.30.60:3000

#Default user
admin

#Default password
admin

Create Dashboard
#

  • Click + > New dashboard
  • Click Add visualization
  • Select Prometheus
  • Paste the query int the Metric box
# Disk space usage in percentage
(node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes * 100

  • Click Run queries

  • Select a virtualization like Pie chart

  • Use the Title box to define a name for the panel, for example “Storage”

  • Click Apply to save the panel

  • Click on the disk symbol to save the dashboard


Links # 

# Download Prometheus
https://prometheus.io/download/

# Download Node Exporter
https://prometheus.io/download/#node_exporter

# Official Documentation
https://prometheus.io/docs/introduction/overview/

# Exporters
https://prometheus.io/docs/instrumenting/exporters/

# Node Exporter
https://prometheus.io/docs/guides/node-exporter/

# Node Exporter GitHub
https://github.com/prometheus/node_exporter