Score:0

webhook MS Teams integration with Prometheus - request failed

us flag

I'm struggling with Microsoft Teams/Prometheus integration on K8s cluster. I used helm to start all components. I have correctly working Prometheus and Alertmanager. It seems that all works fine. Prometheus communicate with Alertmanager. Then prometheus-msteams receives POST alert from Alert Manager and it should send it to a Microsoft Teams Channel but it's not.

2022/03/01 06:49:38 [DEBUG] POST https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
2022/03/01 06:50:08 [ERR] POST https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx request failed: Post https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx: dial tcp 42.12.12.542:443: i/o timeout

30s and timeout. I thought that it may be a proxy issue. So I added extraEnv parameter to config map, restarted pod but nothing changed. So my configuration looks like:

apiVersion: v1
data:
  connectors.yaml: |
    connectors:
      - alertmanager-warning: https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
      - alertmanager-critical: https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx

    extraEnvs:
      HTTPS_PROXY: http://my-proxy.com:911
kind: ConfigMap
metadata:

I also loged into container to check if /etc/config/connectors.yaml is OK. I'm afraid that this extraEnvs doesn't work somehow. From K8s worker node I tried manually (with curl) post some test json to MS Chanel and without proxy it hanged. When I exported HTTPS_PROXY var the message was sucessfully created in MS Teams Chanel.

 export HTTPS_PROXY=http://my-proxy.com:911
 curl -X POST -d @test.json https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx

Do you have any idea what can cause the problem? Is this HTTPS_PROXY env should be listed when I type printvenv in prometheus-msteams container?

Score:0
us flag

Problem solved :) HTTPS_PROXY should be defined deployment not configMap

env:
  - name: HTTPS_PROXY
    value: http://myproxy.com:911
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.