Score:2

Kubernetes ingress 502 bad gateway on DigitalOcean

es flag

I am trying to deploy a NestJS app with Kubernetes on DigitalOcean and I have followed this tutorial, but I am always getting a 502 Bad Gateway from the nginx-ingress-controller.

That's my deployment.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: nestjs-api
spec:
  ports:
    - port: 80
      targetPort: 3001
  selector:
    app: nestjs-api
---
# Create nestjs-api
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nestjs-api
  labels:
    app: nestjs-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nestjs-api
  template:
    metadata:
      labels:
        app: nestjs-api
    spec:
      containers:
        - name: nestjs-api
          image: registry.digitalocean.com/nestjs-registry/nestjs-api
          ports:
            - containerPort: 3001
          envFrom:
            - secretRef:
                name: api-env
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nestjs-ingress
spec:
  rules:
    - host: api.mydomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nestjs-api
                port:
                  number: 80

That's the Docker image on registry.digitalocean.com/nestjs-registry/nestjs-api

FROM node:14-alpine3.14 AS BUILD_IMAGE

RUN apk update && apk add yarn curl bash make && rm -rf /var/cache/apk/*

RUN curl -sfL https://install.goreleaser.com/github.com/tj/node-prune.sh | bash -s -- -b /usr/local/bin

WORKDIR /usr/src/app

# install dependencies
RUN yarn --frozen-lockfile

COPY . .
RUN yarn install
RUN yarn build

RUN npm prune --production

RUN /usr/local/bin/node-prune

FROM node:14-alpine3.14

USER 1000
RUN mkdir -p /home/node/app/
RUN mkdir -p /home/node/app/node_modules
RUN mkdir -p /home/node/app/dist

RUN chown -R 1000:1000 /home/node/app
RUN chown -R 1000:1000 /home/node/app/node_modules
RUN chown -R 1000:1000 /home/node/app/dist

WORKDIR /home/node/app

COPY --from=BUILD_IMAGE /usr/src/app/dist /home/node/app/dist
COPY --from=BUILD_IMAGE /usr/src/app/node_modules /home/node/app/node_modules

EXPOSE 3001
ENTRYPOINT ["node"]
CMD ["/home/node/app/dist/main.js"]

That's the log of my nginx-ingress-controller

2021/09/29 18:37:12 [error] 590#590: *147263 connect() failed (111: Connection refused) while connecting to upstream, client: MY_HOME_IP, server: api.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.229:3001/", host: "api.mydomain.com"
2021/09/29 18:37:12 [error] 590#590: *147263 connect() failed (111: Connection refused) while connecting to upstream, client: MY_HOME_IP, server: api.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.229:3001/", host: "api.mydomain.com"
2021/09/29 18:37:12 [error] 590#590: *147263 connect() failed (111: Connection refused) while connecting to upstream, client: MY_HOME_IP, server: api.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.229:3001/", host: "api.mydomain.com"
MY_HOME_IP - - [29/Sep/2021:18:37:12 +0000] "GET / HTTP/1.1" 502 150 "-" "PostmanRuntime/7.28.4" 204 0.000 [default-nestjs-api-80] [] 10.244.0.229:3001, 10.244.0.229:3001, 10.244.0.229:3001 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 a54bfdae6e0b77bf894e53d8ac8fa29e

Here are some outputs from kubectl

$ kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP             NODE                  NOMINATED NODE   READINESS GATES
nestjs-api-6bcccbdbd5-zmdqg   1/1     Running   0          61m   10.244.0.238   api-wn5e3n2u8-u3j8q   <none>           <none>

$ kubectl get service -o wide
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE    SELECTOR
nestjs-api   ClusterIP   10.245.37.142   <none>        80/TCP    3h     app=nestjs-api
kubernetes     ClusterIP   10.245.0.1      <none>        443/TCP   5d6h   <none>


$ kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS      RESTARTS   AGE     IP             NODE                  NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-psdn2        0/1     Completed   0          6h34m   10.244.0.251   api-wn5e3n2u8-u3j8q   <none>           <none>
ingress-nginx-admission-patch-x8vvs         0/1     Completed   1          6h34m   10.244.0.252   api-wn5e3n2u8-u3j8q   <none>           <none>
ingress-nginx-controller-68649d49b8-bj7vp   1/1     Running     0          6h34m   10.244.0.175   api-wn5e3n2u8-u3j8q   <none>           <none>

UPDATE

The log of my main.js shows that the app is listening on port 3001

{"message":"Application is running on: http://127.0.0.1:3001"}

Which comes from the following code

// Get Server IP and PORT from configuration
const ip = process.env.SERVER_IP;
const port = parseInt(process.env.SERVER_PORT, 10);

// Start server
await app.listen(port, ip);
logger.log(`Application is running on: ${await app.getUrl()}`);
logger.log(`Environment: ${environment}`);

These are the vars saved in api-env as a Secret on k8s.

SERVER_IP: 127.0.0.1
SERVER_PORT: 3001

Anyway as suggested by @mdaniel i have added the livenessProbe to the spec of my Deployment and the Pod crash.

    spec:
      containers:
        - name: nestjs-api
          image: registry.digitalocean.com/nestjs-registry/nestjs-api
          ports:
            - containerPort: 3001
          livenessProbe:
            httpGet:
              port: 3001
              path: '/'
          envFrom:
            - secretRef:
                name: api-env

Now I'm really confused. There is something wrong in my configuration but I have no idea what.

Thanks in advance.

in flag
Hi fasenderos welcome to S.F. The message seems pretty straigthforward -- your `main.js` is evidently not listening on :3001. You can get k8s to help you by adding a `livenessProbe: { httpGet: { port: 3001, path: "/" } }` to the `spec:` and it'll bomb your Pod if node stops listening, making the misconfiguration a lot louder
Mikołaj Głodziak avatar
id flag
Did you try the suggest from mdaniel? Let us know what the result came out.
es flag
Hi mdaniel thanks for reply. I have updated the question.
in flag
You're also really doing yourself and your future colleagues a grave disservice by hiding configuration information in a `Secret` since the server-ip and port are **for sure** not secret, but every interaction with them is going to require a `Role` that can read cluster Secrets and then decoding the secret's fields
Score:2
in flag

{"message":"Application is running on: http://127.0.0.1:3001"}

There's your problem; just like with any server, if you only bind to localhost, nothing except that one "machine" can access that port.

You will want

data:
  SERVER_IP: 0.0.0.0
  SERVER_PORT: '3001'
es flag
Even setting the IP address 0.0.0.0 does not work, but I probably there is something wrong with my Nestjs app. I’m using the Fastify adapter and I have followed their documentation https://docs.nestjs.com/techniques/performance
es flag
Solved, there were a firewall rule that block my app on startup. Anyway, a part the firewall rule, changing the IP to 0.0.0.0 was the key. Thanks.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.