Score:0

502 gateway error on Kubernetes Ingress (K3S - Bare Metal)

ca flag

I have a k3s cluster with a master on a cloud VPS but with 2 remote worker nodes, both on the same network. They're connected via a VPN to the cloud provider. The worker nodes were able to join just fine and were able to create new pods, but it seems that communication is a problem because I can't contact the pods from an Ingress.

When I run my Traefik Ingress (I tried Nginx too, same result), it keeps returning 502 even though the configuration seems correct, so it can't contact the backend. The backend pods are on the remote workers, while the ingress controller is on the VPS.

The backend deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kiwoonapi
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kiwoonapi
  template:
    metadata:
      labels:
        app: kiwoonapi
    spec:
      containers:
      - name: backend
        imagePullPolicy: IfNotPresent
        image: *pullplace*
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: *secret*

The service:

apiVersion: v1
kind: Service
metadata:
  name: kiwoonapisvc
  labels:
    run: kiwoonapi
spec:
  selector:
    app: kiwoonapi
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

The Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kiwoonapi-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: *domain*
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: kiwoonapisvc
            port:
              number: 80
  tls:
    - hosts:
      - *domain*
      secretName: *domain-crt*

Status of the cluster:

kubectl get pods
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-5ff76fc89d-qzmpc   1/1     Running     0          54m
kube-system   metrics-server-86cbb8457f-rhrcf           1/1     Running     0          54m
kube-system   coredns-7448499f4d-xf5lj                  1/1     Running     0          54m
kube-system   helm-install-traefik-crd-q8c28            0/1     Completed   0          54m
kube-system   helm-install-traefik-xd7vp                0/1     Completed   1          54m
kube-system   svclb-traefik-jd8cr                       2/2     Running     0          53m
kube-system   traefik-97b44b794-ht2l4                   1/1     Running     0          53m
kube-system   svclb-traefik-9zcvm                       2/2     Running     0          52m
kube-system   svclb-traefik-zlj7w                       2/2     Running     0          49m
default       kiwoonapi-6c668ffd67-lsbll                1/1     Running     0          49m
default       kiwoonapi-6c668ffd67-csqz9                1/1     Running     0          49m
default       kiwoonapi-6c668ffd67-89x95                1/1     Running     0          49m

kubectl get svc --all-namespaces
NAMESPACE     NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                        PORT(S)                      AGE
default       kubernetes       ClusterIP      10.43.0.1       <none>                             443/TCP                      54m
kube-system   kube-dns         ClusterIP      10.43.0.10      <none>                             53/UDP,53/TCP,9153/TCP       54m
kube-system   metrics-server   ClusterIP      10.43.109.147   <none>                             443/TCP                      54m
kube-system   traefik          LoadBalancer   10.43.176.123   10.0.0.131,10.0.0.98,192.168.0.4   80:30410/TCP,443:32524/TCP   53m
default       kiwoonapisvc     ClusterIP      10.43.96.214    <none>                             80/TCP                       47m

kubectl get ing
NAME                CLASS    HOSTS          ADDRESS                            PORTS     AGE
kiwoonapi-ingress   <none>   *domain*   10.0.0.131,10.0.0.98,192.168.0.4   80, 443   42m

The Dockerfile does expose port 80 and 443. I'm not sure if not including 443 in the deployment somehow affects it.

The LoadBalancer used is not from the cloud: it's something that's built into k3s and essentially piggybacks off an existing node.

Anyone see a problem in the configuration somewhere? I'm able to provide more information if needed.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.