Join Log in
Score:2
Carlos Sosa avatar Server

How to solve error 503 in Kubernetes NGINX Ingress

cn flag
Carlos Sosa

I'm trying to access Kubernetes Dashboard using NGINX INGRESS but for some reason I'm getting a 503 error.

I'm running Kubernetes locally in my macbook with docker desktop.

First thing I did was apply/install NGINX INGRESS CONTROLLER

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.47.0/deploy/static/provider/cloud/deploy.yaml

Second thing I did was to apply/install kubernetes dashboard YML File

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml

Third Step was to apply the ingress service

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |-
      proxy_ssl_server_name on;
      proxy_ssl_name $host;
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: kubernetes-dashboard
                port:
                  number: 433

When I try to access http://localhost and/or https://localhost I get a 503 Service Temporarily Unavailable Error from nginx

Not sure what I'm doing wrong.

Here is part of the log from the NGINX POD

I0630 23:36:42.049398      10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0630 23:36:42.055306      10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"47868", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0630 23:36:42.056435      10 controller.go:146] "Configuration changes detected, backend reload required"
I0630 23:36:42.124850      10 controller.go:163] "Backend successfully reloaded"
I0630 23:36:42.125333      10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [30/Jun/2021:23:36:44 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.003 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.002 400 395aec46af3b21e79cd650f2f86722f3
2021/06/30 23:36:44 [error] 1222#1222: *17477 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
2021/06/30 23:36:45 [error] 1222#1222: *17512 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [30/Jun/2021:23:36:45 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.002 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 a15e1e48987948cb93503b494d188654
2021/07/01 00:09:31 [error] 1224#1224: *49299 recv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:31 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.002 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 ac6b88ca52b73358c39371cb4422761d
2021/07/01 00:09:32 [error] 1221#1221: *49336 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:32 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 2c5cd2d9403a8e50a77fdc897c694792
2021/07/01 00:09:33 [error] 1221#1221: *49338 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.000 400 f1f630c886d20b9b9c59bd9e0e0e3860
2021/07/01 00:09:33 [error] 1224#1224: *49344 recv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 2ab6774dec6e2a89599c4745d24b9661
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.000 400 c9147e08203d9ec8e7b0d0debab8d556
2021/07/01 00:09:33 [error] 1222#1222: *49360 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
I0701 00:10:19.024220      10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:10:19.026772      10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:10:19.027392      10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"50637", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:10:19.102759      10 controller.go:163] "Backend successfully reloaded"
I0701 00:10:19.103246      10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:11:27 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c449f6e8082761ddc3432f956f4701f2
192.168.65.3 - - [01/Jul/2021:00:11:29 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 3a41974b01c5e63e734fce6e37b98e4c
192.168.65.3 - - [01/Jul/2021:00:11:56 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 408 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c01f7bec83d3be6b26703b8808f9922a
192.168.65.3 - - [01/Jul/2021:00:11:58 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 24 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - dc39bcddd4ecfdefe931bf16fe3c1557
192.168.65.3 - - [01/Jul/2021:00:16:36 +0000] "GET / HTTP/1.1" 503 190 "-" "curl/7.64.1" 73 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 82aad4321afbccb3fc54ac75d96b66ee
192.168.65.3 - - [01/Jul/2021:00:31:47 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c4ab3d2f272be4d38df62c0ffd50bfe9
I0701 00:48:02.059067      10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:48:02.062292      10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"53737", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:48:02.062876      10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:48:02.131494      10 controller.go:163] "Backend successfully reloaded"
I0701 00:48:02.131787      10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:48:12 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - d50e3bb0db3a5fa7581c405b8c50d5c8
192.168.65.3 - - [01/Jul/2021:00:48:14 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 15 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c8d8752fb4d79d5bc084839ef9a767b2
I0701 00:49:50.908720      10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:49:50.911044      10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:49:50.911350      10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"53896", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:49:50.979935      10 controller.go:163] "Backend successfully reloaded"
I0701 00:49:50.980213      10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:50:55 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - d62a8012bc23bbc35a47621d54d68a62
192.168.65.3 - - [01/Jul/2021:00:51:00 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 15 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 0cbfd2274ad687fc1aaff76dbc483659

Any help would be greatly appreciated.

3247
1 + 0
Score:2
Dawid Kruk avatar Server
cn flag
Dawid Kruk

I've reproduced this setup and encountered the same issue as described in the question:

  • 503 Service Temporarily Unavailable Error

Focusing specifically on this setup, to fix above error you will need to modify the part of your Ingress manifest:

  • from:
                name: kubernetes-dashboard
                port:
                  number: 433
  • to:
                name: kubernetes-dashboard
                port:
                  number: 443 # <-- HERE!

You've encountered the 503 error as nginx was sending a request to a port that was not hosting the dashboard (433 -> 443).

After that change, I was fortunate enough to see the Dashboard login page.

I'd also recommend you following a guide to create a user that could connect to the dashboard with it's bearer token:

Additional resources:

0 + 1
Carlos Sosa avatar
cn flag
Carlos Sosa
Unbelievable, thanks a million for catching that typo! You saved me a lot of headaches!.. Thanks again!
2
Reply
Score:0
Z.Liu avatar Server
br flag
Z.Liu

from the official doc,

It's important the file generated is named auth (actually - that the secret has a key data.auth), otherwise the ingress-controller returns a 503

which means the secret need have a auth key

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.