Score:0

prometheus alert for Kubernetes misfiring

gb flag

On a K8s cluster I have Prometheus Operator and AlertManager running.

I have this alert to catch incidents when a critical pod is down:

 - alert: KubernetesContainerMission-gslNotRunning
    expr: kube_pod_status_ready{condition="false", pod=~"mission-gsl.*"} == 1 OR on() vector(0)
    for: 5m
    labels:
      severity: warning
      environment: PRODUCTION_ENV
    annotations:
      summary: CUSTOM mission-gsl pod not running for more than 5min (instance {{ $labels.instance }})
      description: "mission-gsl pod not running for more than 5min"

This deployment gets automatically rolling-restarted on a schedule every hour, and stays down for maybe 30sec during the process.

I would expect the alert to not fire, since I'm stipulating a 5min down period, yet it does.

What am I missing?

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.