DEV Community

Deepanshu Anand
Deepanshu Anand

Posted on

Alerts and Alert Manager in Prometheus

#monitoring #alertmanager #prometheu #gmail

In the last blog we walked through the process of setting up a Prometheus server with Grafana dashboard. In this blog we will be setting up the alertmanager in the prometheus server.

Structure of an alertmanager and the prometheus server

alertmanager1

Defining rules for prometheus server

  • In the last blog we created a alerts.rules config files, this file will contain all the rules for our prometheus server 
cd /opt/prometheus/ vim alerts.rules
Enter fullscreen mode Exit fullscreen mode 
groups: - name: Linux-disk-space rules: - alert: Linux-DiskSpaceAlert expr: (node_filesystem_avail_bytes/node_filesystem_size_bytes)*100 < 20 for: 5m annotations: summary: "Disk space is running low" description: 'The available disk space on {{ $labels.instance }} is less than 80percent' - name: Linux-memory-utilisation rules: - alert: Linux-MemoryUsage expr: ((node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100 ) < 20 for: 2m annotations: Summary: "High RAM Usage" description: 'Memory usage on {{ $labels.instance }} is greater than 80percent' - name: Linux-up rules: - alert: Linux-up-status expr: up{job='node-exporter'} < 1 for: 2m annotations: Summary: "Linux machine/exporter down" description: '{{ $labels.instance }} metrics could not be fetched machine may be down' - name: Windows-memory-utilisation rules: - alert: Windows-MemoryUsage expr: ((windows_os_physical_memory_free_bytes/windows_cs_physical_memory_bytes)*100 ) < 20 for: 2m annotations: Summary: "High RAM Usage" description: 'RAM usage on {{ $labels.instance }} is higher than 80 percent' - name: Windows-diskspace-utilisation rules: - alert: Windows-diskspace-utilisation expr: (windows_logical_disk_free_bytes/windows_logical_disk_size_bytes)*100 < 20 for: 2m annotations: Summary: "Disk Space running low" description: 'Diskspace utilisation on {{ $labels.instance }} is higher than 80 percent' - name: Windows-up rules: - alert: Windows-up-status expr: up{job='win-exporter'} < 1 for: 2m annotations: Summary: "Windows machine/exporter down" description: '{{ $labels.instance }} metrics could not be fetched machine may be down'
Enter fullscreen mode Exit fullscreen mode

Setting the alertmanager

  • for this walkthrough we will be setting up gmail alerts using gmail smtp server

1. setting up gmail smtp server

To set up a gmail smtp server for prometheus we need a gmail account say sender@gmail.com on which 2-step verification is enabled

step-1

step-2

step-3

  • your app password is your secret-key which will be used in alertmanager.yml as auth_password step-4

2. config file for alertmanager 

cd /opt/prometheus/ touch alertmanager.yml vim alertmanager.yml
Enter fullscreen mode Exit fullscreen mode 
global: resolve_timeout: 1m route: receiver: 'gmail-alert-notification' group_wait: 10s group_interval: 1m repeat_interval: 1m # sender@gmail is the gmail account on which you have generated your app password receivers: - name: 'gmail-alert-notification' email_configs: - to: 'receiver@gmail.com' from: 'sender@gmail.com' smarthost: smtp.gmail.com:587 auth_username: 'sender@gmail.com' auth_identity: 'sender@gmail.com' auth_password: 'secret-key'
Enter fullscreen mode Exit fullscreen mode

3. installing the alertmanager docker image

We will be installing the docker image of alertmanager on the same machine on which we installed the prometheus server

docker run -d -p 9093:9093 -v /opt/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml --restart unless-stopped --name alertmanager prom/alertmanager
Enter fullscreen mode Exit fullscreen mode
  • verify that alertmanager is running by visiting alertmanager-ip:9093 in your browser

Adding rules and alertmanager in the prometheus 

cd /opt/prometheus/ vim prometheus.yml
Enter fullscreen mode Exit fullscreen mode 
global: scrape_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'node-exporter' static_configs: - targets: ['ubuntu_id:9100'] - job_name: 'win-exporter' static_configs: - targets: ['windows_ip:9182'] rule_files: - "/etc/prometheus/alerts.rules" alerting: alertmanagers: - static_configs: - targets: - "ubuntu_ip:9093" # ubuntu_ip is the IP on which the alertmanager container is installed, use localcost:9093 if same as prometheus server
Enter fullscreen mode Exit fullscreen mode 
# if prometheus docker is already running run the following command docker restart prometheus # if not run the following docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus
Enter fullscreen mode Exit fullscreen mode

Your prometheus and alertmanager is ready and can be verified by visiting prometheus_ip:9090 and alertmanager_ip:9093 on your browser

I hope this blog helped you understand how to setup prometheus with alertmanager. Stay connected for more such blogs

Top comments (0)