I'm trying to deploy Grafana in Azure Kubernetes service (AKS), through Helm. I'm using the Helm charts provided by Bitnami. I've been following this Microsoft tutorial to set up the application gateway ingress controller.
The pods are running, and I have a ClusterIP
service in front of them:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-grafana ClusterIP 10.0.230.64 <none> 3000/TCP 29m
Output of kubectl describe svc
:
Name: my-grafana
Namespace: default
Labels: app.kubernetes.io/component=grafana
app.kubernetes.io/instance=my-grafana
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=grafana
helm.sh/chart=grafana-8.4.5
Annotations: meta.helm.sh/release-name: my-grafana
meta.helm.sh/release-namespace: default
service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector: app.kubernetes.io/component=grafana,app.kubernetes.io/instance=my-grafana,app.kubernetes.io/name=grafana
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.0.168.200
IPs: 10.0.168.200
Port: http 3000/TCP
TargetPort: dashboard/TCP
Endpoints: 10.244.0.20:3000,10.244.0.21:3000
Session Affinity: None
Events: <none>
Next, I also have an ingress defined, which automatically configures the Application Gateway (AGW) using AGIC (AGW and AKS cluster are defined in the same resource group).
NAME CLASS HOSTS ADDRESS PORTS AGE
my-grafana <none> * 52.XXX.XXX.XXX 80 29m
Output of kubectl describe ing
:
Name: my-grafana
Labels: app.kubernetes.io/component=grafana
app.kubernetes.io/instance=my-grafana
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=grafana
helm.sh/chart=grafana-8.4.5
Namespace: default
Address: 52.XXX.XXX.XXX
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/* my-grafana:http (10.244.0.16:3000,10.244.0.17:3000)
Annotations: kubernetes.io/ingress.class: azure/application-gateway
meta.helm.sh/release-name: my-grafana
meta.helm.sh/release-namespace: default
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PortResolutionError 31m (x2 over 32m) azure/application-gateway Code="ErrorServiceResolvedToInvalidPort" Message="service default/my-grafana and service port http was resolved to an invalid service-backend port 65536, defaulting to port 80"
If I try to connect to the external address (public IP attached to the AGW), I get an HTTP 502 error with the above configuration.
If I change hostname: "*"
to hostname: grafana.local
in the values.yaml
file, the public IP now returns an HTTP 404 error instead.
When I'm connected to one of the pods, I can connect to Grafana using the cluster IP as well as each pod individually, on port 3000.
As far as I can tell, the issue is between the AGW and the Ingress/Service, but I have no idea what to investigate further. Any hints would be very much appreciated!