Injected vault-agent pod failing to start, api server & vault aren't communicating

I have a local kubernetes cluster using kind. It is a single node cluster.

On this cluster I am following this guide to setup Vault & the vault-agent-injector.

If I follow the tutorial step by step the end result is that the orgchart pod will fail to start as it is still waiting for the vault-agent sidecar to start.

kubectl get pods:

NAMESPACE            NAME                                         READY   STATUS      RESTARTS   AGE
default              dnstools                                     0/1     Completed   0          22m
default              orgchart-7457f8489d-hxtqt                    1/1     Running     0          26m
default              orgchart-798cbc6c76-4kzrl                    0/2     Init:0/1    0          25m
default              vault-0                                      1/1     Running     0          28m
default              vault-agent-injector-79d479cf7d-kz67q        1/1     Running     0          28m
kube-system          coredns-558bd4d5db-gnxl7                     1/1     Running     0          45m
kube-system          coredns-558bd4d5db-ph95m                     1/1     Running     0          45m
kube-system          etcd-kind-control-plane                      1/1     Running     0          45m
kube-system          kindnet-bjvzg                                1/1     Running     0          45m
kube-system          kube-apiserver-kind-control-plane            1/1     Running     0          45m
kube-system          kube-controller-manager-kind-control-plane   1/1     Running     0          45m
kube-system          kube-proxy-v5545                             1/1     Running     0          45m
kube-system          kube-scheduler-kind-control-plane            1/1     Running     0          45m
local-path-storage   local-path-provisioner-547f784dff-mp79j      1/1     Running     0          45m

kubectl get svc:

    NAMESPACE     NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes                 ClusterIP   10.96.0.1      <none>        443/TCP                  44m
default       vault                      ClusterIP   10.96.42.180   <none>        8200/TCP,8201/TCP        27m
default       vault-agent-injector-svc   ClusterIP   10.96.156.79   <none>        443/TCP                  27m
default       vault-internal             ClusterIP   None           <none>        8200/TCP,8201/TCP        27m
kube-system   kube-dns                   ClusterIP   10.96.0.10     <none>        53/UDP,53/TCP,9153/TCP   44m

kubectl describe org:

Name:         orgchart-798cbc6c76-4kzrl
Namespace:    default
Priority:     0
Node:         kind-control-plane/172.18.0.2
Start Time:   Fri, 20 Aug 2021 13:07:13 +0000
Labels:       app=orgchart
              pod-template-hash=798cbc6c76
Annotations:  vault.hashicorp.com/agent-inject: true
              vault.hashicorp.com/agent-inject-secret-database-config.txt: internal/data/database/config
              vault.hashicorp.com/agent-inject-status: injected
              vault.hashicorp.com/role: internal-app
Status:       Pending
IP:           10.244.0.12
IPs:
  IP:           10.244.0.12
Controlled By:  ReplicaSet/orgchart-798cbc6c76
Init Containers:
  vault-agent-init:
    Container ID:  containerd://d7dc0c9d089a1e59d85ac16a422a5195f5e7026a5c98a004d5e6a6a576900119
    Image:         hashicorp/vault:1.8.0
    Image ID:      docker.io/hashicorp/vault@sha256:54d12f84c0bfbed57d2d2e1d89099540c5ccb4e6c3b2b0a123b82dab12c99e75
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Fri, 20 Aug 2021 13:07:14 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      <REDACTED>
    Mounts:
      /home/vault from home-init (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx52m (ro)
      /vault/secrets from vault-secrets (rw)
Containers:
  orgchart:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx52m (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent:
    Container ID:
    Image:         hashicorp/vault:1.8.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      <REDACTED>
    Mounts:
      /home/vault from home-sidecar (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx52m (ro)
      /vault/secrets from vault-secrets (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-lx52m:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  home-init:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  home-sidecar:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  vault-secrets:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Normal   Scheduled         26m                 default-scheduler  Successfully assigned default/orgchart-798cbc6c76-4kzrl to kind-control-plane
  Normal   Pulled            26m                 kubelet            Container image "hashicorp/vault:1.8.0" already present on machine
  Normal   Created           26m                 kubelet            Created container vault-agent-init
  Normal   Started           26m                 kubelet            Started container vault-agent-init
  Warning  DNSConfigForming  53s (x25 over 26m)  kubelet            Search Line limits were exceeded, some search paths have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local <REDACTED> <REDACTED> <REDACTED>

Looking in the apiserver logs I see the following message:

E0820 13:03:55.315142       1 dispatcher.go:171] failed calling webhook "vault.hashicorp.com": Post "https://vault-agent-injector-svc.default.svc:443/mutate?timeout=10s": dial tcp 10.96.156.79:443: connect: no route to host

And looking in the vault-agent-injector logs I see the following message:

2021-08-20T13:03:55.887Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2021-08-20T13:03:55.896Z [INFO]  handler.auto-tls: Generated CA
2021-08-20T13:03:55.899Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-08-20T13:05:55.808Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-08-20T13:07:13.447Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-08-20T13:10:16.613Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s

I have configured my kind cluster to be able use MutatingAdmissionWebhooks:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
  extraPortMappings:
  - containerPort: 80
    hostPort: 8080
    protocol: TCP
  - containerPort: 443
    hostPort: 8443
    protocol: TCP
networking:
  apiServerAddress: "127.0.0.1"
  apiServerPort: 443

What next steps can I take to help me debug the issue?

I am thinking this is an issue with the dns lookup for the vault-agent-injector-svc but am not sure where to start.

Thanks, Max Sargent