Score:0

HTTP 502/404 with Grafana in Kubernetes, using AGIC

us flag

I'm trying to deploy Grafana in Azure Kubernetes service (AKS), through Helm. I'm using the Helm charts provided by Bitnami. I've been following this Microsoft tutorial to set up the application gateway ingress controller.

The pods are running, and I have a ClusterIP service in front of them:

NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
my-grafana   ClusterIP   10.0.230.64   <none>        3000/TCP   29m

Output of kubectl describe svc:

Name:              my-grafana
Namespace:         default
Labels:            app.kubernetes.io/component=grafana
                   app.kubernetes.io/instance=my-grafana
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=grafana
                   helm.sh/chart=grafana-8.4.5
Annotations:       meta.helm.sh/release-name: my-grafana
                   meta.helm.sh/release-namespace: default
                   service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector:          app.kubernetes.io/component=grafana,app.kubernetes.io/instance=my-grafana,app.kubernetes.io/name=grafana
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.0.168.200
IPs:               10.0.168.200
Port:              http  3000/TCP
TargetPort:        dashboard/TCP
Endpoints:         10.244.0.20:3000,10.244.0.21:3000
Session Affinity:  None
Events:            <none>

Next, I also have an ingress defined, which automatically configures the Application Gateway (AGW) using AGIC (AGW and AKS cluster are defined in the same resource group).

NAME         CLASS    HOSTS   ADDRESS         PORTS   AGE
my-grafana   <none>   *       52.XXX.XXX.XXX   80      29m

Output of kubectl describe ing:

Name:             my-grafana
Labels:           app.kubernetes.io/component=grafana
                  app.kubernetes.io/instance=my-grafana
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=grafana
                  helm.sh/chart=grafana-8.4.5
Namespace:        default
Address:          52.XXX.XXX.XXX
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /*   my-grafana:http (10.244.0.16:3000,10.244.0.17:3000)
Annotations:  kubernetes.io/ingress.class: azure/application-gateway
              meta.helm.sh/release-name: my-grafana
              meta.helm.sh/release-namespace: default
Events:
  Type     Reason               Age                From                       Message
  ----     ------               ----               ----                       -------
  Warning  PortResolutionError  31m (x2 over 32m)  azure/application-gateway  Code="ErrorServiceResolvedToInvalidPort" Message="service default/my-grafana and service port http was resolved to an invalid service-backend port 65536, defaulting to port 80"

If I try to connect to the external address (public IP attached to the AGW), I get an HTTP 502 error with the above configuration.

If I change hostname: "*" to hostname: grafana.local in the values.yaml file, the public IP now returns an HTTP 404 error instead.

When I'm connected to one of the pods, I can connect to Grafana using the cluster IP as well as each pod individually, on port 3000.

As far as I can tell, the issue is between the AGW and the Ingress/Service, but I have no idea what to investigate further. Any hints would be very much appreciated!

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.