Score:2

Rancher x509 Certificate Expired yet not

ec flag

When running kubectl, I get the error

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2021-10-05T11:59:14-04:00 is after 2021-09-29T19:21:40Z

So clearly it says the cert is expired. Only problem is I'm not sure which cert it is.

I've checked

  • HAProxy (Rancher sits behind L7 HAProxy with LE cert)
  • Certs in the secrets shown from sudo k3s kubectl get secrets -n cattle-system
  • Certs in /etc/kubernetes/ssl on the K8s node

All are fine (not expired), as this particular rancher/k8s instance was brought up in June, so all the certs are only a few months old, and expire either 1 year or 10 years later.

So what cert is expired that needs to be updated?

Some information about my setup:

  • Rancher 2.5.9 HA (K3s v1.21.1+k3s1) (single-node, Ubuntu 20.04)
  • Kubernetes 1.20.9-rancher1-1 (single-node, Control plane/Worker/etcd, Ubuntu 20.04)
in flag
Have you checked the cert that is usually embedded in your kubeconfig? And have you actually _looked_ at the cert being returned, as in `echo '' | openssl s_client -servername whatever.example.com -showcerts -connect whatever.example.com:6443 | openssl x509 -noout -text` kind of deal?
ec flag
@mdaniel yes, they are all valid. And I just did. The cert returned by that command is `/etc/kubernetes/ssl/kube-apiserver.pem`.
in flag
@cclloyd Have you found the solution?
ec flag
@Adam the LE cert was invalid because one of the root certs was invalid. Had to update root cert packages on host systems and regenerated cert.
Andrew Skorkin avatar
tr flag
Hi @cclloyd Could you post an answer, since it would be better for other users to see that you found the solution and also for indexing the answer by the site?
Score:1
tr flag

This is a community wiki answer posted for better visibility. Feel free to expand it.

Based on information from comments

Root cause:

One of the root certificates is invalid. This caused the Let's Encrypt certificate to be invalid.

Solution:

  1. Update root cert packages on host systems
  2. Regenerate certificate
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.