Score:0

Disabling email for Prometheus Alertmanager based on severity

ru flag

We are monitoring a HPC computing cluster using a combination of Prometheus, Alertmanager and Grafana. On our machines, things like SWAP memory filling up to essentially the limit happen frequently, and while it is useful to see the corresponding info-level alerts in the Grafana Alerts dashboard, we would preferably not send the corresponding emails.

Is there a way to mute/disable all, say, alerting emails that have severity info in the alertmanager.yml config file?

The alerts are all defined similar to this one (adjusted from https://awesome-prometheus-alerts.grep.to/rules.html):

  - alert: HostSwapIsFillingUp
    expr: (1 - (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes)) * 100 > 95
    for: 60m
    labels:
      severity: info
    annotations:
      summary: Host swap is filling up (instance {{ $labels.instance }})
      description: "Swap is filling up (>95%)\n  VALUE = {{ $value }}"

and the corresponding section in the alertmanager.yml file reads

  routes:
    - match:
        severity: 'warning'
      repeat_interval: 24h
      continue: true
    - match:
        severity: 'info'
      repeat_interval: 24h
      continue: true
      receiver: dropped

receivers:
  - name: 'admin-mails'
    email_configs:
      - to: 'admins@DOMAIN'
  - name: 'dropped'
    email_configs:
      - to: 'admins@DOMAIN'

Is there a possibility to make sure that the info-level alerts never cause emails while simultaneously still having them "fire", so that Grafana will display them?

Score:0
ru flag

After some trial-and-error, the following seems to do the trick:

  routes:
    - match:
        severity: 'warning'
      repeat_interval: 24h
      continue: true
    - match:
        severity: 'info'
      repeat_interval: 24h
      continue: true
      receiver: dropped

receivers:
  - name: 'admin-mails'
    email_configs:
      - to: 'admins@DOMAIN'
  - name: 'dropped'

So the idea is to simply have the receiver unconfigured. With this, no emails are generated anymore, but the alerts are still shown in Grafana.

I will leave this here in case someone else encounters the same question.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.