Icinga2 Dockerized - Basic Setup for Icinga2 Monitoring

Table of Contents

In this tutorial I set up a basic Icinga2 Setup that is managed with Icinga Director. Here are the links to the original repository:
https://hub.docker.com/r/jordan/icinga2/
https://github.com/jjethwa/icinga2

In my Tutorial Setup the IP addresses and hostnames are as follows:

Icinga Server:  192.168.30.195 ubnticinga
Host:           192.168.30.196 ubnts1

Incinga2
#

Setup
#

Note:
Don’t use special characters like “!$?” when you define the passwords for MYSQL_ROOT_PASSWORD and DEFAULT_MYSQL_PASS otherwise the Icinga Director Daemon want start!

# Clone Repository
git clone https://github.com/jjethwa/icinga2.git

# Change into Directory
cd icinga2

# Define MySQL Root PW
echo 'MYSQL_ROOT_PASSWORD=vCVxtKTavGrjtTep' > secrets_sql.env

# Uncomment and define Admin User & PW
- ICINGAWEB2_ADMIN_USER=jkw
- ICINGAWEB2_ADMIN_PASS=test123!

# Define MySQL User & PW, otherwise Container wan't restart
- DEFAULT_MYSQL_PASS=dakGSXxaXAunXj2p
- DEFAULT_MYSQL_USER=icinga2

The final Docker Compose file should look like this:

version: '2'
services:
  icinga2:
    #image: jordan/icinga2
    build:
      context: ./
      dockerfile: Dockerfile
    restart: on-failure:5
    # Set your hostname to the FQDN under which your
    # sattelites will reach this container
    hostname: ubnticinga
    env_file:
      - secrets_sql.env
    environment:
      #- APACHE2_HTTP=REDIRECT
      - ICINGA2_FEATURE_GRAPHITE=1
      # Important:
      # keep the hostname graphite the same as
      # the name of the graphite docker-container
      - ICINGA2_FEATURE_GRAPHITE_HOST=graphite
      - ICINGA2_FEATURE_GRAPHITE_PORT=2003
      - ICINGA2_FEATURE_GRAPHITE_URL=http://graphite
      # - ICINGA2_FEATURE_GRAPHITE_SEND_THRESHOLDS=true
      # - ICINGA2_FEATURE_GRAPHITE_SEND_METADATA=false
      - ICINGAWEB2_ADMIN_USER=jkw
      - ICINGAWEB2_ADMIN_PASS=test123
      #- ICINGA2_USER_FULLNAME=Icinga2 Docker Monitoring Instance
      - DEFAULT_MYSQL_HOST=mysql
      - DEFAULT_MYSQL_PASS=dakGSXxaXAunXj2p # Define MySQL PW, otherwise Container wan't restart
      - DEFAULT_MYSQL_USER=icinga2 # Define MySQL User, otherwise Container wan't restart
    volumes:
      - ./data/icinga/cache:/var/cache/icinga2
      - ./data/icinga/certs:/etc/apache2/ssl
      - ./data/icinga/etc/icinga2:/etc/icinga2
      - ./data/icinga/etc/icingaweb2:/etc/icingaweb2
      - ./data/icinga/lib/icinga:/var/lib/icinga2
      - ./data/icinga/lib/php/sessions:/var/lib/php/sessions
      - ./data/icinga/log/apache2:/var/log/apache2
      - ./data/icinga/log/icinga2:/var/log/icinga2
      - ./data/icinga/log/icingaweb2:/var/log/icingaweb2
      - ./data/icinga/log/mysql:/var/log/mysql
      - ./data/icinga/spool:/var/spool/icinga2
      - ./data/icinga/cron.d/icinga:/etc/cron.d/icinga
      # Sending e-mail
      #  See: https://github.com/jjethwa/icinga2#sending-notification-mails
      #  If you want to enable outbound e-mail, edit the file mstmp/msmtprc
      #  and configure to your corresponding mail setup. The default is a
      #  Gmail example but msmtp can be used for any MTA configuration.
      #  Change the aliases in msmtp/aliases to your recipients.
      #  Then uncomment the rows below
      # - ./msmtp/msmtprc:/etc/msmtprc:ro
      # - ./msmtp/aliases:/etc/aliases:ro
    ports:
      - "80:80"
      - "443:443"
      - "5665:5665"
  graphite:
    image: graphiteapp/graphite-statsd:latest
    container_name: graphite
    restart: on-failure:5
    hostname: graphite
    volumes:
      - ./data/graphite/conf:/opt/graphite/conf
      - ./data/graphite/storage:/opt/graphite/storage
      - ./data/graphite/log/graphite:/var/log/graphite
      - ./data/graphite/log/carbon:/var/log/carbon
  mysql:
    image: mariadb
    container_name: mysql
    restart: always
    env_file:
      - secrets_sql.env
    environment:
      - MARIADB_AUTO_UPGRADE=1
    volumes:
      - ./data/mysql/data:/var/lib/mysql
      # If you have previously used the container's internal DB use:
      #- ./data/icinga/lib/mysql:/var/lib/mysql

Start Icinga
#

# Start Icinga2
docker compose up -d

# Check Logs
docker compose logs

Docke PS Output:

9f281ce77d5a    mysql
e10b7bc2bdbf    graphite
6a3bc2c5afc4    icinga2-icinga2-1

Icinga Web Interface
#

Live Grid:
http://192.168.30.195/icingaweb2/monitoring/list/servicegrid?limit=100%2C100

Check Icinga Version:
http://192.168.30.195/icingaweb2/about

Additional Configurtion:
http://192.168.30.195/icingaweb2/navigation/dashboard?name=configuration

SSL/TSL Encryption
#

In this tutorial I use an local VM to host my Icinga2 Server. In order to enable SSL/TSL encryption I spin up an EC2 instance, point the subdomain “icinga2.jklug.work” from Route 53 to the public IP of the EC2 instance and request a certificate with Certbot. After that I copy the certificates to the local VM and enable the redirect http to https option for Apache. The last step is to create a local DNS entry on the Client that points to the VM with Icinga2.

Create Certificates and copy them to the Icinga2 Instance:

# Install Cerbot
sudo apt install certbot

# Request Certificate
sudo certbot certonly --standalone -d icinga2.jklug.work

# Certificate Directory (copy to local VM)
/etc/letsencrypt/archive/icinga2.jklug.work

# Copy and rename Certificates (copy from local VM directory)
sudo cp ./cert1.pem ~/icinga2/data/icinga/certs/icinga2.crt &&
sudo cp ./privkey1.pem ~/icinga2/data/icinga/certs/icinga2.key

Enable http to https Redirection:

# Uncomment Apache Redirect in Docker Compose file
environment:
  - APACHE2_HTTP=REDIRECT

# Restart Docker Container
cd ~/icinga2
docker compose down
docker compose up -d

Create DNS entry on local host:

# Open hosts file (Start Notepad as Administrator)
C:\Windows\System32\drivers\etc\hosts

# Add local DNS entry
192.168.30.195 icinga2.jklug.work

Test the HTTPS Encryption:

Client Monitoring
#

Install necessary dependencies
#

# Ubuntu Repository
sudo apt update && sudo apt -y install apt-transport-https wget gnupg

sudo su

wget -O - https://packages.icinga.com/icinga.key | gpg --dearmor -o /usr/share/keyrings/icinga-archive-keyring.gpg

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
 echo "deb [signed-by=/usr/share/keyrings/icinga-archive-keyring.gpg] https://packages.icinga.com/ubuntu icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src [signed-by=/usr/share/keyrings/icinga-archive-keyring.gpg] https://packages.icinga.com/ubuntu icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

# Update Package Manager
sudo apt update

# Install Icinga2 and Monitoring Plugins
sudo apt install icinga2 monitoring-plugins -y

Node Wizzard: Connection to Master
#

# Run Node Wizzard
sudo icinga2 node wizard

Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]: # enter

Please specify the common name (CN) [ubnts1]: # ubnts1

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): #ubnticinga

Do you want to establish a connection to the parent node from this node? [Y/n]: # y

Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): # 192.168.30.195

Master/Satellite endpoint port [5665]: # enter
Add more master/satellite endpoints? [y/N]: # enter

Is this information correct? [y/N]: # y

Please specify the request ticket generated on your Icinga 2 master (optional).
(Hint: icinga2 pki ticket --cn 'ubnts1'):

Switch to Icinga2 Server

# Open Container Terminal
docker exec -it icinga2-icinga2-1 bash

# Run the following Command
icinga2 pki ticket --cn 'ubnts1'

# Copy Shell Output
3cbee4db60efcd79c56c14961ce93290c8c2b548

Move back to node Wizzard

Please specify the request ticket generated on your Icinga 2 master (optional).
(Hint: icinga2 pki ticket --cn 'ubnts1'): # 5cf6daae1ca438a3e12a6de29397193694bbd96b

Please specify the API bind host/port (optional):
Bind Host []: # enter
Bind Port []: # enter

Accept config from parent node? [y/N]: # y
Accept commands from parent node? [y/N]: # y

Reconfiguring Icinga...
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.

Local zone name [ubnts1]: # enter
Parent zone name [master]: # enter

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: # enter

Do you want to disable the inclusion of the conf.d directory [Y/n]: # enter

Restart Icinga2 Agent on host

# Restart Icinga2 Agent
sudo systemctl restart icinga2

# Check Status
sudo systemctl status icinga2

# Check Logs
tail -f /var/log/icinga2/icinga2.log

Node Wizzard: No Connection to Master
#

# Run Node Wizzard
sudo icinga2 node wizard

Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]: # Enter

Please specify the common name (CN) [ubnts1]: # ubnts1

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): # ubnticinga

Do you want to establish a connection to the parent node from this node? [Y/n]: # n

Please copy the public CA certificate from your master/satellite
into '/var/lib/icinga2/certs//ca.crt' before starting Icinga 2.
Please specify the API bind host/port (optional):
Bind Host []: # Enter
Bind Port []: # Enter

Accept config from parent node? [y/N]:  # y
Accept commands from parent node? [y/N]: # y
Reconfiguring Icinga...

Local zone name [gilgamesh.thesiestagroup.com]: # Enter
Parent zone name [master]: # Enter

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: # Enter

Do you want to disable the inclusion of the conf.d directory [Y/n]: # Enter
Disabling the inclusion of the conf.d directory...

Done.
Now restart your Icinga 2 daemon to finish the installation!

Copy cert to host or import manually

# Copy cert to host
sudo vi /var/lib/icinga2/certs/ca.crt
# Set permissions
sudo chown nagios:nagios /var/lib/icinga2/certs/ca.crt

Add new host with Icinga2 Director
Activity log: deploy 1 pending changes
Restart Icinga2 Agent on host

# Restart Icinga2 Agent
sudo systemctl restart icinga2

# Check Status
sudo systemctl status icinga2

# Check Logs
tail -f /var/log/icinga2/icinga2.log

Sign Certificate

# Exec Internal Terminal
docker exec -it icinga2-icinga2-1 bash

# List Certificates
icinga2 ca list

Shell output:
-----------------------------------------------------------------|--------------------------|--------|--------
8c4a9f0e3c1236d78c789e4a5d69fdef2b3a4e5b6c7d8e9fb0a1234567890abc | Oct 19 13:46:58 2023 GMT |        | CN = ubnts1

# Sign Certificate
icinga2 ca sign 8c4a9f0e3c1236d78c789e4a5d69fdef2b3a4e5b6c7d8e9fb0a1234567890abc

Icinga Director
#

Host and Host Group
#

Create Host Group
#

Go to: Icinga Director / Hosts / Host Groups: Add

Click: Add, Store & Deploy

Create Host Template
#

Go to: Icinga Director / Hosts / Host Templates: Add

Add Host to Host Group
#

Go to: Icinga Director / Hosts / Hosts: Add

Service Sets and Templates
#

Create Service Templates
#

Go To: Icinga Director / Services / Service Templates: Add

The Service Templates we’ll create are based on the External Commands that are already available with Icinga. We just define the specific parameters for the Checks in the Templates.

Create Service Set
#

Go to: Icinga Director / Services / Service Sets: Add

Define the Host Group for which the Service Set applies:

Go to the “Services” section of the Service Set and add the Service Templates we created: apt, ping & load

Service Grid
#

Open the Icinga Serive Grid, it now should look like this:

Custom Commands
#

This is an Example for an Custom Command that checks if the Linux Server needs a reboot.

Add Script on Client
#

# Open Plugin Directory
cd /usr/lib/nagios/plugins

# Create File for Command
vi check_reboot.sh

# Make File executable
chmod +x check_reboot.sh

Add the following Script:

#!/bin/bash
# Copyright (C) 2017 Martin Scharm <https://binfalse.de/contact/>
#
# This file is part of bf-monitoring.
# <https://binfalse.de/software/nagios/>
# <https://github.com/binfalse/monitoring>
# 
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.


# this script checks if the file /var/run/reboot-required is present
# should work on all debian-based systems...


if [ -f "/var/run/reboot-required" ]
then
    echo "reboot required!"
    exit 1
else
    echo "looks good over here..."
    exit 0
fi

Create Custom Command
#

Go to: Icinga Director / Commands / Commands: Add

Define the Custom Command and add the path to the script:

# Add path to script
/usr/lib/nagios/plugins/check_reboot.sh

Create Service Template
#

Go to: Icinga Director / Services / Service Templates: Add

Add the new created “linux_reboot” Service Template to the “JKW-Debian” Service Set we created earlier and check the Service Grid :)

Incinga2 #

Setup #

Start Icinga #

Icinga Web Interface #

SSL/TSL Encryption #

Client Monitoring #

Install necessary dependencies #

Node Wizzard: Connection to Master #

Node Wizzard: No Connection to Master #

Icinga Director #

Host and Host Group #

Create Host Group #

Create Host Template #

Add Host to Host Group #

Service Sets and Templates #

Create Service Templates #

Create Service Set #

Service Grid #

Custom Commands #

Add Script on Client #

Create Custom Command #

Create Service Template #