Score:0

kubernetes cluster cert-manager in crashloopbackoff

pm flag

I had a kubernetes cluster running on a raspberry pi that had a simple server running on it. It's a small project i'm using to learn kubernetes.

After coming back to the project i realised it wasn't responding anymore. I Assumed this was because the https certificate needed renewing. But i cant figure out how to renew the certificate while pods aren't running.

I thought it would just be a case of reapplying the cluster-issuer but obviously that wouldn't work if some pods aren't running/being created.

trying to apply the cluser-issuer.yml gives me-

Error from server (InternalError): error when creating "cluster-issuer.yml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.100.140.135:443: connect: connection refused

i can also see this when getting cert-manager pods-

cert-manager   cert-manager-7fb78674d7-tnjf9              1/1     Running            4 (21h ago)       25h
cert-manager   cert-manager-cainjector-5dfc946d84-7wxrl   0/1     Error              271 (5m51s ago)   25h
cert-manager   cert-manager-webhook-8744b7588-zvmps       0/1     CrashLoopBackOff   270 (107s ago)    25h

output of kubectl logs cert-manager-cainjector-5dfc946d84-7wxrl -n cert-manager-

Error: error creating manager: Get "https://10.96.0.1:443/api?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

server-deployment.yml-

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: spoonobi/multi-server-arm
          ports:
            - containerPort: 8888

server-cluster-ip-service.yml-

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 8888
      targetPort: 8888

ingress-service.yml-

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: letsencrypt-prod # tell ingress to use https
    nginx.ingress.kubernetes.io/ssl-redirect: 'true' # redirect from http to https
spec:
  tls:
    - hosts:
        - ecmatrials.com
        - www.ecmatrials.com
      secretName: secret-ecmatrials-com
  rules:
    - host: ecmatrials.com
      http:
        paths:
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 8888
    - host: www.ecmatrials.com
      http:
        paths:
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 8888

cluster-issuer.yml-

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-redacted
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

service-nodeports.yml-

apiVersion: v1
kind: Service
metadata:
  name: server-nodeports
spec:
  type: NodePort
  selector:
    component: server
  ports:
    - name: http
      port: 80
      targetPort: 8888
      nodePort: 30602
    - name: https
      port: 443
      targetPort: 8888
      nodePort: 30824
in flag
Hi lloyd noone welcome to S.F. It's almost certainly due to a misconfigured CNI, as that 10.96.0.1 is a special `Service` IP that corresponds to the kubernetes API in every Namespace in the cluster. No one can _guess_ from your limited information what specifically is wrong with your cluster, but consider [editing your question](https://serverfault.com/posts/1119353/edit) to include the output of `kubectl get po -A` in order to see what the status is of the control plane and any CNI pods. Good luck!
lloyd noone avatar
pm flag
hi @mdaniel, at some point when i was debugging and trying different things the coredns pod also stopped running. Found out this was due to old docker images still being referenced so deleted those. Then when i fixed that i managed to get the web hook and api-server etc running again after kubeadm reset and flushing iptables. So you were right about CNI i guess. Shame i had to go with the nuclear option.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.