AWS EKS Ingress Timeout On Any Non-Root Path

We have configured an Ingress resource on our EKS cluster with rewrites from /.* on the load balancer to the matching URI upstream. If we visit staging.my-domain.com/, we see a successful health-check response as expected. However, any other url, e.g. /api/, results in a timeout from the load balancer. Below is the configuration. (SSL is disabled for now while we get it figured out). Any help would be appreciated!

# Ingress Controller: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/
# YAML: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/alb-ingress.md
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    external-dns.alpha.kubernetes.io/hostname: staging.my-domain.com
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: nlx-api
spec:
  rules:
  - host: staging.my-domain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: our-api
            port:
              number: 80
---

apiVersion: v1
kind: Service
metadata:
  name: our-api
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  type: LoadBalancer
  selector:
    app: our-api

It looks like your Rewrite Target is wrong. Look at this general example from documentation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation. For example, the ingress definition above will result in the following rewrites:

• rewrite.bar.com/something rewrites to rewrite.bar.com/
• rewrite.bar.com/something/ rewrites to rewrite.bar.com/
• rewrite.bar.com/something/new rewrites to rewrite.bar.com/new

In your situation, if you try to access staging.my-domain.com/, you are rewrited to the same address. Everything is fine. But you can rewrite only this address. You should change your manifest like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    external-dns.alpha.kubernetes.io/hostname: staging.my-domain.com
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
  name: nlx-api
spec:
  rules:
  - host: staging.my-domain.com
    http:
      paths:
      - path: /(.*)
        pathType: Prefix
        backend:
          service:
            name: our-api
            port:
              number: 80

In this situation any characters captured by (.*) will be assigned to the placeholder $1(this is first capture group), which is then used as a parameter in the rewrite-target annotation.

This is the configuration that ended up working:

# Ingress Controller: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/
# YAML: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/alb-ingress.md
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
    external-dns.alpha.kubernetes.io/hostname: example.com
    nginx.ingress.kubernetes.io/rewrite-target: /$1 # this is where the problem was
  name: nlx-api
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: / # this was correct
        pathType: Prefix
        backend:
          service:
            name: nlx-api
            port:
              number: 80