Score:0

kubernetes cert-manager https attempt propagation check failed

pm flag

Im trying to set up https on a local bare metal kubernetes cluster. Im getting an

cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" error.

I think the way i've set up my nodeports is the issue but not sure on the fix. Only think i can think of is adding http://.com/.well-known/acme-challenge/ as a route but obviously this isn't a long term solution.

ingress-service.yaml-

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    # nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: letsencrypt-prod # tell ingress to use https
    # nginx.ingress.kubernetes.io/ssl-redirect: 'true' # redirect from http to https
spec:
  tls:
    - hosts:
        - <domain>.com
        - www.<domain>.com
      secretName: secret-redacted-com
  rules:
    - host: <domain>.com
      http:
        paths:
          - path: /?(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 8888
    - host: www.<domain>.com
      http:
        paths:
          - path: /?(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 8888

cluster-issuer.yaml-

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-redacted-prod
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
         # class: nginx
          ingressTemplate:
            metadata:
              annotations:
                kubernetes.io/ingress.class: nginx

server-cluster-ip-service.yaml-

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 8888
      targetPort: 8888

server-deployment.yaml-

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: spoonobi/multi-server-arm
          ports:
            - containerPort: 8888

service-nodeports.yaml-

apiVersion: v1
kind: Service
metadata:
  name: server-nodeports
spec:
  type: NodePort
  selector:
    component: server
  ports:
    - name: http
      port: 80
      targetPort: 8888
      nodePort: 30602
    - name: https   
      port: 443
      targetPort: 8888
      nodePort: 30824

logs-

kubectl logs cert-manager-7fb78674d7-8l8v4 -n cert-manager

kubectl logs cert-manager-7fb78674d7-8l8v4 -n cert-manager
I0109 14:21:46.934907       1 start.go:75] cert-manager "msg"="starting controller" "git-commit"="a96bae172ddb1fcd4b57f1859ab9d1a9e94f7451" "version"="v1.10.1"
I0109 14:21:46.935260       1 controller.go:242] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.96.0.10:53"]
W0109 14:21:46.942385       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0109 14:21:47.020721       1 controller.go:70] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]" 
I0109 14:21:47.026062       1 controller.go:134] cert-manager/controller "msg"="starting leader election" 
I0109 14:21:47.028151       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-controller...
I0109 14:21:47.029212       1 controller.go:91] cert-manager/controller "msg"="starting metrics server" "address"={"IP":"::","Port":9402,"Zone":""}
I0109 14:21:47.164717       1 leaderelection.go:258] successfully acquired lease kube-system/cert-manager-controller
I0109 14:21:47.189500       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca"
I0109 14:21:47.190713       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-venafi"
I0109 14:21:47.192837       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="gateway-shim"
I0109 14:21:47.193391       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="orders"
I0109 14:21:47.193503       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager"
I0109 14:21:47.205713       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver"
I0109 14:21:47.208892       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi"
I0109 14:21:47.213435       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-ca"
I0109 14:21:47.213533       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-selfsigned"
I0109 14:21:47.216346       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness"
I0109 14:21:47.218932       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim"
I0109 14:21:47.221861       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme"
I0109 14:21:47.234795       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager"
I0109 14:21:47.241310       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-acme"
I0109 14:21:47.242004       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger"
I0109 14:21:47.256862       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="issuers"
I0109 14:21:47.257295       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned"
I0109 14:21:47.257529       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing"
I0109 14:21:47.266041       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager"
I0109 14:21:47.267554       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-vault"
I0109 14:21:47.267654       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-metrics"
I0109 14:21:47.269328       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers"
I0109 14:21:47.385919       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="challenges"
I0109 14:21:47.393223       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault"
I0109 14:21:47.470499       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-secret-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
I0109 14:21:47.490596       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ecmatrials.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2rzhm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-78145582" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.490592       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ecmatrials.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-prw2b" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-665678090" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.491386       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ecmatrials.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-ctsfv" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-665678090" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.491727       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ecmatrials.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-mhrwj" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-665678090" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.491392       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ecmatrials.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-q29l7" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-78145582" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.493451       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ecmatrials.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-chbv4" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-78145582" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0109 14:21:48.011116       1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://ecmatrials.com/.well-known/acme-challenge/hjygDO-DEv4upt89EGUSa4GrYVvf5489zsN21aXITv4': Get \"http://ecmatrials.com/.well-known/acme-challenge/hjygDO-DEv4upt89EGUSa4GrYVvf5489zsN21aXITv4\": dial tcp 88.98.208.82:80: connect: connection refused" "dnsName"="ecmatrials.com" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-78145582" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
kubectl logs cert-manager-cainjector-5dfc946d84-scg8x -n cert-manager
I0109 14:21:42.567868       1 start.go:126] "starting" version="v1.10.1" revision="a96bae172ddb1fcd4b57f1859ab9d1a9e94f7451"
I0109 14:21:42.876693       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I0109 14:22:56.469583       1 leaderelection.go:258] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I0109 14:22:56.470249       1 recorder.go:103] cert-manager/events "msg"="cert-manager-cainjector-5dfc946d84-scg8x_9520c889-77e0-4bcb-9786-c018eaa01ea8 became leader" "object"={"kind":"Lease","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"230a9367-b459-444f-b54c-4cfba19c00c2","apiVersion":"coordination.k8s.io/v1","resourceVersion":"26612"} "reason"="LeaderElection" "type"="Normal"
I0109 14:22:56.674060       1 controller.go:185] cert-manager/certificate/mutatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.MutatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} []}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.674053       1 controller.go:185] cert-manager/certificate/validatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.ValidatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} []}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.674107       1 controller.go:185] cert-manager/certificate/apiservice "msg"="Starting EventSource" "source"="&{{%!s(*v1.APIService=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil>   false [] 0 0} {[]}}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.675248       1 controller.go:185] cert-manager/certificate/apiservice "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil>   <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.675321       1 controller.go:185] cert-manager/certificate/apiservice "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.675363       1 controller.go:193] cert-manager/certificate/apiservice "msg"="Starting Controller" 
I0109 14:22:56.675447       1 controller.go:185] cert-manager/certificate/validatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil>   <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.675664       1 controller.go:185] cert-manager/certificate/validatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.674419       1 controller.go:185] cert-manager/secret/customresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.CustomResourceDefinition=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} { {  []   []}  [] <nil> false} {[] {  []   []} []}}) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.675947       1 controller.go:185] cert-manager/secret/customresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.675981       1 controller.go:193] cert-manager/secret/customresourcedefinition "msg"="Starting Controller" 
I0109 14:22:56.675837       1 controller.go:193] cert-manager/certificate/validatingwebhookconfiguration "msg"="Starting Controller" 
I0109 14:22:56.674858       1 controller.go:185] cert-manager/certificate/mutatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil>   <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.677760       1 controller.go:185] cert-manager/certificate/mutatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.677828       1 controller.go:193] cert-manager/certificate/mutatingwebhookconfiguration "msg"="Starting Controller" 
I0109 14:22:56.679318       1 controller.go:185] cert-manager/certificate/customresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.CustomResourceDefinition=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} { {  []   []}  [] <nil> false} {[] {  []   []} []}}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.680910       1 controller.go:185] cert-manager/certificate/customresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil>   <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.679983       1 controller.go:185] cert-manager/secret/apiservice "msg"="Starting EventSource" "source"="&{{%!s(*v1.APIService=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil>   false [] 0 0} {[]}}) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.680215       1 controller.go:185] cert-manager/secret/validatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.ValidatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} []}) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.680453       1 controller.go:185] cert-manager/secret/mutatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.MutatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} []}) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.682127       1 controller.go:185] cert-manager/certificate/customresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x4000110460}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.682880       1 controller.go:185] cert-manager/secret/apiservice "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.683512       1 controller.go:193] cert-manager/secret/apiservice "msg"="Starting Controller" 
I0109 14:22:56.683766       1 controller.go:185] cert-manager/secret/validatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.683829       1 controller.go:193] cert-manager/secret/validatingwebhookconfiguration "msg"="Starting Controller" 
I0109 14:22:56.683996       1 controller.go:185] cert-manager/secret/mutatingwebhookconfiguration "msg"="Starting EventSource" "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0x400011c260}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0109 14:22:56.684046       1 controller.go:193] cert-manager/secret/mutatingwebhookconfiguration "msg"="Starting Controller" 
I0109 14:22:56.684144       1 controller.go:193] cert-manager/certificate/customresourcedefinition "msg"="Starting Controller" 
I0109 14:22:56.778634       1 controller.go:227] cert-manager/secret/customresourcedefinition "msg"="Starting workers" "worker count"=1
I0109 14:22:56.779696       1 controller.go:227] cert-manager/certificate/apiservice "msg"="Starting workers" "worker count"=1
I0109 14:22:56.780597       1 controller.go:227] cert-manager/certificate/validatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0109 14:22:56.781039       1 controller.go:227] cert-manager/certificate/mutatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0109 14:22:56.785983       1 controller.go:227] cert-manager/secret/apiservice "msg"="Starting workers" "worker count"=1
I0109 14:22:56.786830       1 controller.go:227] cert-manager/secret/mutatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0109 14:22:56.789745       1 controller.go:227] cert-manager/certificate/customresourcedefinition "msg"="Starting workers" "worker count"=1
I0109 14:22:56.793102       1 controller.go:227] cert-manager/secret/validatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0109 14:22:56.810090       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0109 14:22:56.813377       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0109 14:22:56.821140       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0109 14:22:56.823988       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
kubectl logs cert-manager-webhook-8744b7588-sgq6p -n cert-manager
I0109 14:21:44.799137       1 feature_gate.go:245] feature gates: &{map[]}
W0109 14:21:44.800367       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0109 14:21:45.252258       1 webhook.go:129] cert-manager "msg"="using dynamic certificate generating using CA stored in Secret resource" "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I0109 14:21:45.255076       1 server.go:133] cert-manager/webhook "msg"="listening for insecure healthz connections" "address"=":6080"
I0109 14:21:45.268500       1 server.go:197] cert-manager/webhook "msg"="listening for secure connections" "address"=":10250"
I0109 14:21:46.298072       1 dynamic_source.go:266] cert-manager/webhook "msg"="Updated cert-manager webhook TLS certificate" "DNSNames"=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]

in flag
Well, what was in the logs for their request? Have you already increased the verbosity of cert-manager? Are you running a modern version? No one can possibly **guess** what's wrong with your setup; you'll have to show what troubleshooting you've already tried and why that's not helping you. Good luck
lloyd noone avatar
pm flag
Hi @mdaniel, i've added more logs for more info including requests. Ive nuked this cluster and started over with fresh installs of everything so i dont think versions should be the issue. Ive tried adding the /.wellknown/ route to my actual ingress as it doesnt seem like requests are going to the acme solver from what i can tell but that didnt help.
in flag
Heh, no, I mean quite literally `kubectl logs $pod` for the `cert-manager` pods and also the nginx pods, showing the 404 that it received. We, collectively, need to understand where the misalignment came from. Also, I can save you quite a bit of trouble "redacting" your super sekrit domain name, since you leaked it multiple times but ironically left the ACME key in place. Don't bother, we're not going to p0wn your sekrit domain name
lloyd noone avatar
pm flag
haha fair enough. Ive added logs from the cert-manager pods. I dont have any nginx pods as such but my ingress is annotated to use nginx. Bit confused there. The domain does work on public internet but not the get to /.well-known.
in flag
_I dont have any nginx pods as such_ well, there's your problem; solve **that** first
lloyd noone avatar
pm flag
Im confused now, my ingress has nginx annotation and the actual ingress-service does everything it needs to do and nothing gets flagged.
lloyd noone avatar
pm flag
Oh actually, i think the first time i deployed this, i used ingress-nginx which sets up a nodeport. Whereas this time, after nuking it, i just saw the nodeport was gone and set up my own nodeport. Guess i need to install ingress-nginx and use the nodeport that its sets up.
lloyd noone avatar
pm flag
@mdaniel, ah damn that was it. Immediately started working after applying the bare-metal ingress-nginx yaml from their documentation and deleting my own nodeport. And using the right port numbers obvs. I really should write some documentation for myself for future reference. Thanks a lot! major facepalm though haha.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.