Score:1

NGINX Ingress on Kubernetes doesn't use HTTPS

in flag

I am setting a Kubernetes cluster on bare metal. I used Kubeadm for the installation. To make my services accessible from outside the cluster, I installed an NGINX Ingress, using the following documentation : NGINX doc

Because I don't want to communicate with my services without TLS security, I configured the certificate thanks to cert-manager : Cert-manager doc.

$kubectl get certificate
NAME                   READY   SECRET            AGE
certificate-tls-prod   True    tls-secret-prod   3h1m
tls-secret-prod        True    tls-secret-prod   41m

The certificate generation works and are recognized by the client.

Now, I am trying to install my own Docker registry (registry:2.7.1) in the cluster. After deploying the Pod and the Service, I am trying to do a docker login from my client, but it doesn't work.

Ingress configuration :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    cert-manager.io/issuer: "letsencrypt-prod"
  namespace: default
  name: nginx-ingress
spec:
  tls:
  - hosts:
    - example.com
    secretName: tls-secret-prod
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: docker-registry
            port: 
              number: 5000

When I try to reach the service doing a simple curl :

curl -v https://example.com/

*   Trying <cluster-ip>:443...
* Connected to example.com (<cluster-ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=example.com
*  start date: Jul 13 10:03:45 2021 GMT
*  expire date: Oct 11 10:03:44 2021 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.77.0
> Accept: */*
> 
* Received HTTP/0.9 when not allowed
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (1) Received HTTP/0.9 when not allowed

The handshake seems correct. But, something's wrong in the end.

Log of the Docker registry pod :

http: TLS handshake error from <ingress-pod-ip>:38686: tls: first record does not look like a TLS handshake

Log of the Ingress Pod :

[error] 84#84: *40 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: <cluster-ip>, server: example.com, request: "GET / HTTP/1.1", upstream: "http://<registry-pod-ip>:5000/", host: "example.com"

Log of Docker login from the client:

docker login https://example.com
Error response from daemon: Get "https://example.com/v2/": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

From what I understand, the problem comes from the fact that one of the component try to communicate with http instead of https. By the way, the upsteam is in http, which seems strange.
Unfortunately, I can't seem to fix this problem. I tried several annotations (which are commented) in the ingress declaration to force the use of https, but it didn't work.

I saw this issue on Github, but my apiserver doesn't have the parameter --kubelet-https=false

I would appreciate it if you had any clue that could help me.

djdomi avatar
za flag
could it be that you are using http instead of https?
Thomas avatar
in flag
@djdomi What do you mean ? What part of my configuration do you think I could be wrong with?
Mikołaj Głodziak avatar
id flag
Which version of Kuberneted did you use? Please keep in mind that there are 3 types of Nginx. Open Source Nginx Ingress Controller, Nginx Incorporaton (nginx inc) and Nginx Incorporaton Plus. Try to use open source nginx. [Here is the example](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/)?
Thomas avatar
in flag
Thank you for your help ! I managed to make it works. It seems that it came from the docker registry configuration. I modified the certs and restarted the pod.
Score:1
id flag

As Thomas has mentioned in the comment:

I managed to make it works. It seems that it came from the docker registry configuration. I modified the certs and restarted the pod.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.