Score:1

Grafana does not send notifications through reverse proxy

in flag

I am using Grafana on a server that does not have Internet, I need to send Telegram notifications, for that I am using Nginx server in another machine.

Server A:

  • Grafana installed / Running on port 3000.
  • Grafana url: www.example.grafana.com
  • IP of the server (example): 10.16.216.122
  • It uses this webhook to send telegram messages: api.telegram.org
  • I also configured /etc/hosts to send that notifications to my server:
#/etc/hosts
10.16.216.16  api.telegram.org

Server B:

  • IP of the server (example): 10.16.216.16
  • Nginx installed, with this config:
server {
    listen 80;
    listen [::]:80 http2 ssl;

    server_name http://10.16.216.122;
    location / {
        proxy_pass http://10.16.216.122:3000/;
    }}

When I try to send the notification, I receive this in the logs:

t=2021-10-27T17:10:07+0200 lvl=eror msg="Failed to send webhook" logger=alerting.notifier.telegram error="Post https://api.telegram.org/botXXXXXXXXXX/sendMessage: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" webhook=TelegramTest
t=2021-10-27T17:10:07+0200 lvl=eror msg="failed to send notification" logger=alerting.notifier uid= error="Post https://api.telegram.org/botXXXXXXXXXXXXXX/sendMessage: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
t=2021-10-27T17:10:07+0200 lvl=eror msg="failed to send notification" logger=alerting.notifier uid= error="Post https://api.telegram.org/botXXXXXXXXXXXXXXXXXXX/sendMessage: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
t=2021-10-27T17:10:07+0200 lvl=eror msg="Failed to send alert notifications" logger=context userId=51 orgId=5 uname=XXXX error="Post https://api.telegram.org/botXXXXXXXXXXXXX/sendMessage: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
t=2021-10-27T17:10:07+0200 lvl=eror msg="Request Completed" logger=context userId=51 orgId=5 uname=XXXXX method=POST path=/api/alert-notifications/test status=500 remote_addr=10.26.216.122 time_ms=30006 size=48 referer=http://10.26.216.122/alerting/notification/1/edit

Am I missing something in my configuration on Nginx? Does the Server A need something more to work?

Score:1
ar flag

Webhooks is a outgoing connection from grafana. The reverse proxy handles incoming connections.

If you want to proxy outgoing connections as well, you have to configure a normal, forward proxy, and configure Grafana to use this for outgoing requests.

aldegalan avatar
in flag
Could you please put an example about how to config that?
vidarlo avatar
ar flag
https://www.alibabacloud.com/blog/how-to-use-nginx-as-an-https-forward-proxy-server_595799 first hit googling for 'nginx forward proxy'.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.