I am setting a Kubernetes cluster on bare metal. I used Kubeadm for the installation. To make my services accessible from outside the cluster, I installed an NGINX Ingress, using the following documentation : NGINX doc
Because I don't want to communicate with my services without TLS security, I configured the certificate thanks to cert-manager : Cert-manager doc.
$kubectl get certificate
NAME READY SECRET AGE
certificate-tls-prod True tls-secret-prod 3h1m
tls-secret-prod True tls-secret-prod 41m
The certificate generation works and are recognized by the client.
Now, I am trying to install my own Docker registry (registry:2.7.1) in the cluster. After deploying the Pod and the Service, I am trying to do a docker login from my client, but it doesn't work.
Ingress configuration :
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# nginx.ingress.kubernetes.io/ssl-redirect: "true"
# nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
# nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
cert-manager.io/issuer: "letsencrypt-prod"
namespace: default
name: nginx-ingress
spec:
tls:
- hosts:
- example.com
secretName: tls-secret-prod
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: docker-registry
port:
number: 5000
When I try to reach the service doing a simple curl :
curl -v https://example.com/
* Trying <cluster-ip>:443...
* Connected to example.com (<cluster-ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=example.com
* start date: Jul 13 10:03:45 2021 GMT
* expire date: Oct 11 10:03:44 2021 GMT
* subjectAltName: host "example.com" matched cert's "example.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.77.0
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (1) Received HTTP/0.9 when not allowed
The handshake seems correct. But, something's wrong in the end.
Log of the Docker registry pod :
http: TLS handshake error from <ingress-pod-ip>:38686: tls: first record does not look like a TLS handshake
Log of the Ingress Pod :
[error] 84#84: *40 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: <cluster-ip>, server: example.com, request: "GET / HTTP/1.1", upstream: "http://<registry-pod-ip>:5000/", host: "example.com"
Log of Docker login from the client:
docker login https://example.com
Error response from daemon: Get "https://example.com/v2/": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
From what I understand, the problem comes from the fact that one of the component try to communicate with http instead of https. By the way, the upsteam is in http, which seems strange.
Unfortunately, I can't seem to fix this problem. I tried several annotations (which are commented) in the ingress declaration to force the use of https, but it didn't work.
I saw this issue on Github, but my apiserver doesn't have the parameter --kubelet-https=false
I would appreciate it if you had any clue that could help me.
