Kubernetes Cert-Manager Expired Certificates

I inherited a GKE Kubernetes environment and have been trying to figure this out for days but unfortunately just don't know what to try next.

The cluster is setup to use cert-manager (installed via helm) to apply Let's Encrypt certificates to the cluster. For some reason, this has worked perfectly for over two years but starting on 4/16 I started seeing SSL warnings in browsers for all notes on the cluster.

When I run kubectl describe certificates site-cloud-tls the certificate seems to have renewed but is not being applied to the ingress traffic.

Name:         site-cloud-tls
Namespace:    cs
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-06-02T09:55:05Z
  Generation:          34
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  cs-nginx
    UID:                   7f312326-851c-11e9-8bf0-4201ac10000c
  Resource Version:        541365011
  UID:                     7f36cc40-851c-11e9-8bf0-4201ac10000c
Spec:
  Dns Names:
    site.cloud (changed name but is correct)
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-dns
  Secret Name:  site-cloud-tls
Status:
  Conditions:
    Last Transition Time:  2022-04-24T05:26:13Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2022-06-15T17:01:48Z
Events:                    <none>

kubectl describe ingress
Name:             cs-nginx
Namespace:        cs
Address:          192.168.1.32
Default backend:  default-http-backend:80 (10.16.3.12:8080)
TLS:
  site-cloud-tls terminates site.cloud (changed naming but seems correct)
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  site.cloud   
                             /   site:8080 (10.10.10.10:8080)

Annotations:                 certmanager.k8s.io/cluster-issuer: letsencrypt-dns
                             kubernetes.io/ingress.class: nginx
                             nginx.ingress.kubernetes.io/ssl-redirect: true
                             nginx.org/websocket-services: datahub
Events:                      <none>

We do have a staging environment which was also affected. I have tried re-installing cert-manager, re-installing nginx-ingress but unfortunately haven't been able to get things back up and running (likely due to a configuration error I've made).

After 3 days, I don't know which way is up and don't know Kubernetes well enough to know what to try next. Any guidance? Can I provide any additional info that might help?

Thank you!

The issue here is that you are referring to a cluster-issuer kind in your Ingress definition:

Annotations:                 certmanager.k8s.io/cluster-issuer: letsencrypt-dns

But the object you have defined is a Certificate kind:

Name:         site-cloud-tls
Namespace:    cs
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate

That is why it is not being applied to the Ingress. What you need is to create an Issuer resource in Kubernetes to handle the certificate. Here you can find an example of a basic ACME ClusterIssuer manifest file:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: [email protected]
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: example-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx