Prometheus - Telegram notification with Alert Manager
Alertmanager is a standalone component that focuses on handling alerts generated by Prometheus. It acts as a centralized alerting system that receives, deduplicates, groups, and sends out notifications.
In this post we will utilize this module to send alert to Telegram based on the rule that we create on Prometheus.
Setting up Alerting rule on Prometheus
The way we create our alerting tool is by detecting the parameter “up”, where if the returned data is 1 it means the node is up and if it is 0 then the node is down.
Now we create a new alerting rule
1
sudo nano /etc/prometheus/alert_server_down.yml
Paste in these lines of configuration
1
2
3
4
5
6
7
8
9
10
groups:
- name: server_down
rules:
- alert: server_down
expr: up == 0
for: 1m
labels:
severity: page
annotations:
summary: Server is down
We give it name “server_down” with the expression “up == 0”, which means if there’s a node with the parameter 0, this rule will be triggered.
We also give it the wait time of 1 minute before firing the rule.
Next we add this new rule to our Prometheus configuration file
1
sudo nano /etc/prometheus/prometheus.yml
Modify the “rule_files” with the yml file that we just created.
1
2
3
4
5
6
7
8
9
10
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "alert_server_down.yml"
Add the newly created “alert_server_down.yml” file below the rule_files.
After that restart the Prometheus service.
1
sudo systemctl restart prometheus
Now on the alert page, you’ll see the “server_down” rule.
This rule by itself doesn’t do anything, it only shows a visual indicator stating a node is down. If we want to trigger an action, we need Alert Manager.
Setting up Alert Manager
First, download the Alert Manager and extract it.
1
2
wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
tar xvzf alertmanager-0.25.0.linux-amd64.tar.gz
Next, navigate to the folder and copy the files to its supposed directories
1
2
3
cd alertmanager-0.25.0.linux-amd64/
sudo cp alertmanager amtool /usr/local/bin/
sudo cp alertmanager.yml /etc/alertmanager/
Set the owner
1
sudo chown -R prometheus:prometheus /etc/alertmanager/alertmanager.yml /usr/local/bin/{alertmanager,amtool}
Configure the Alert Manager yaml file
1
sudo nano /etc/alertmanager/alertmanager.yml
Paste in these lines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 24h
receiver: 'telegram'
receivers:
- name: 'telegram'
webhook_configs:
- url: "http://localhost:8080/alerts"
send_resolved: true
The url is the telegram webhook relay that we will run on a docker later
Run this command to make sure your config file doesn’t have any error
1
amtool check-config /etc/alertmanager/alertmanager.yml
1
2
3
4
5
6
7
Checking '/etc/alertmanager/alertmanager.yml' SUCCESS
Found:
- global config
- route
- 0 inhibit rules
- 1 receivers
- 0 templates
Then we create the alertmanager service file
1
sudo nano /etc/systemd/system/alertmanager.service
Paste in these lines
1
2
3
4
5
6
7
8
9
10
11
12
13
[Unit]
Description=AlertManager Server Service
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml
[Install]
WantedBy=multi-user.target
Finally start the service
1
2
3
sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
Also restart the prometheus service
1
sudo systemctl restart prometheus
Check the status by running this
1
2
sudo systemctl status alertmanager
sudo systemctl status prometheus
1
2
3
4
5
6
7
8
9
● alertmanager.service - AlertManager Server Service
Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-07-18 11:01:14 UTC; 11min ago
Main PID: 5835 (alertmanager)
Tasks: 12 (limit: 9386)
Memory: 14.1M
CPU: 1.175s
CGroup: /system.slice/alertmanager.service
└─5835 /usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml
You can also access the Web UI on here http://198.18.0.201:9093
This page will not show any rule until the rule is triggered.
Installing Docker
Skip this section if you already have docker installed.
Install docker on ubuntu with this command
1
2
sudo snap refresh
sudo snap install docker
Pulling the Docker image
After docker is installed, pull this Alert Manager Telegram Relay docker image.
1
sudo docker pull ghcr.io/janw/alertmanager-telegram:edge
Run this command to make sure the image is present on your machine
1
sudo docker images
1
2
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/janw/alertmanager-telegram edge 49047627b4f6 7 weeks ago 159MB
Run this command to spin up the docker container.
1
2
3
4
5
6
sudo docker run -d --rm \
--name alertmanager-telegram \
-e TELEGRAM_CHAT_ID="{chat-id}" \
-e TELEGRAM_TOKEN="{bot-id}" \
-p 8080:8080 \
ghcr.io/janw/alertmanager-telegram:edge
Use your own chat-id and bot-id from the telegram bot that you have created
Lastly, run this command to check if the container is running
1
sudo docker ps
1
2
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
86be69939b3e ghcr.io/janw/alertmanager-telegram:edge "gunicorn alertmanag…" 32 minutes ago Up 32 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp alertmanager-telegram
Curl localhost:8080 to make sure the web is running
1
curl localhost:8080
1
2
3
4
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
It’ll show an 404 error, That’s normal. All we need to check is the web on that port is up.
And everything is done here. Let’s proceed with testing.
Testing the Alert Manager
Try shutting down one of the monitored node, it’ll be visible on the alert page on Prometheus
We can also see it on Alert Manager
And on the telegram side, we should see our bot sending the alert as well.
