Score:0

Deployment not able to spin the hashicorp vault linked containers, how to fix that?

ke flag

I have followed the steps mentioned in the link

Came till the deployment part, last before step.

After deployment, the pods status is stuck at creation.

kubectl get pods --watch
NAME                                READY   STATUS     RESTARTS   AGE
devwebapp                           0/2     Init:0/1   0          2m11s
nginx-6799fc88d8-9xnqv              1/1     Running    1          98m
vault-0                             1/1     Running    0          25m
vault-agent-injector-c5f9f8-zcv6q   1/1     Running    0          25m

So I ran the describe command and found nothing.

osboxes@osboxes:~$ kubectl describe pod devwebapp
Name:         devwebapp
Namespace:    default
Priority:     0
Node:         uday1-control-plane/172.19.0.2
Start Time:   Tue, 27 Jul 2021 15:50:56 -0400
Labels:       app=devwebapp
Annotations:  vault.hashicorp.com/agent-inject: true
              vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/data/martwebapp/config
              vault.hashicorp.com/agent-inject-status: injected
              vault.hashicorp.com/role: martweb-app
Status:       Pending
IP:           10.244.0.10
IPs:
  IP:  10.244.0.10
Init Containers:
  vault-agent-init:
    Container ID:  containerd://a125495c63dc63e605146b9dd67d1e0e731c43c28e4130156d261efca2aaf54c
    Image:         vault:1.7.3
    Image ID:      docker.io/library/vault@sha256:6085e96fa42c2524eef7bf9af0cf5199da0b16964003b3f88e2b8195b6acb52b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Tue, 27 Jul 2021 15:50:57 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoibWFydHdlYi1hcHAifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjp0cnVlLCJwaWRfZmlsZSI6Ii9ob21lL3ZhdWx0Ly5waWQiLCJ2YXVsdCI6eyJhZGRyZXNzIjoiaHR0cDovL3ZhdWx0LmRlZmF1bHQuc3ZjOjgyMDAifSwidGVtcGxhdGUiOlt7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMvY3JlZGVudGlhbHMudHh0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcInNlY3JldC9kYXRhL21hcnR3ZWJhcHAvY29uZmlnXCIgfX17eyByYW5nZSAkaywgJHYgOj0gLkRhdGEgfX17eyAkayB9fToge3sgJHYgfX1cbnt7IGVuZCB9fXt7IGVuZCB9fSIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9XX0=
    Mounts:
      /home/vault from home-init (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
Containers:
  devwebapp:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent:
    Container ID:
    Image:         vault:1.7.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoibWFydHdlYi1hcHAifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjpmYWxzZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHA6Ly92YXVsdC5kZWZhdWx0LnN2Yzo4MjAwIn0sInRlbXBsYXRlIjpbeyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL2NyZWRlbnRpYWxzLnR4dCIsImNvbnRlbnRzIjoie3sgd2l0aCBzZWNyZXQgXCJzZWNyZXQvZGF0YS9tYXJ0d2ViYXBwL2NvbmZpZ1wiIH19e3sgcmFuZ2UgJGssICR2IDo9IC5EYXRhIH19e3sgJGsgfX06IHt7ICR2IH19XG57eyBlbmQgfX17eyBlbmQgfX0iLCJsZWZ0X2RlbGltaXRlciI6Int7IiwicmlnaHRfZGVsaW1pdGVyIjoifX0ifV19
    Mounts:
      /home/vault from home-sidecar (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-9k5qp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  home-init:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  home-sidecar:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  vault-secrets:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  7m24s  default-scheduler  Successfully assigned default/devwebapp to uday1-control-plane
  Normal  Pulled     7m24s  kubelet            Container image "vault:1.7.3" already present on machine
  Normal  Created    7m24s  kubelet            Created container vault-agent-init
  Normal  Started    7m23s  kubelet            Started container vault-agent-init
osboxes@osboxes:~$ kubectl logs devwebapp -c vault-agent-init
==> Vault agent started! Log data will stream in below:
2021-07-27T19:50:57.835Z [INFO]  sink.file: creating file sink
2021-07-27T19:50:57.836Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-07-27T19:50:57.837Z [INFO]  template.server: starting template server
[INFO] (runner) creating new runner (dry: false, once: false)

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.7.3
             Version Sha: 5d517c864c8f10385bf65627891bc7ef55f5e827

[INFO] (runner) creating watcher
2021-07-27T19:50:57.844Z [INFO]  sink.server: starting sink server
2021-07-27T19:50:57.844Z [INFO]  auth.handler: starting auth handler
2021-07-27T19:50:57.845Z [INFO]  auth.handler: authenticating
2021-07-27T19:51:57.847Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2021-07-27T19:51:58.847Z [INFO]  auth.handler: authenticating
2021-07-27T19:52:58.851Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.91s
2021-07-27T19:53:00.774Z [INFO]  auth.handler: authenticating
2021-07-27T19:54:00.789Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=2.93s
2021-07-27T19:54:03.723Z [INFO]  auth.handler: authenticating
2021-07-27T19:55:03.724Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4.87s
2021-07-27T19:55:08.595Z [INFO]  auth.handler: authenticating
2021-07-27T19:56:09.043Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=8.89s
2021-07-27T19:56:17.940Z [INFO]  auth.handler: authenticating
2021-07-27T19:57:17.942Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=13.86s
2021-07-27T19:57:31.811Z [INFO]  auth.handler: authenticating
2021-07-27T19:58:31.813Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=21.7s
2021-07-27T19:58:53.516Z [INFO]  auth.handler: authenticating
2021-07-27T19:59:53.521Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=39.73s
2021-07-27T20:00:33.254Z [INFO]  auth.handler: authenticating
2021-07-27T20:01:33.255Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1m7.23s
2021-07-27T20:02:40.492Z [INFO]  auth.handler: authenticating
2021-07-27T20:03:40.493Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=2m7.25s
2021-07-27T20:05:47.752Z [INFO]  auth.handler: authenticating
2021-07-27T20:06:47.756Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=3m42.25

Any suggestion on how to resolve this?

Wytrzymały Wiktor avatar
it flag
Hello @uday. Have you tried solutions suggested [here](https://discuss.hashicorp.com/t/kubernetes-vault-agent-init-sidecar-error-context-deadline-exceeded/24570/2) and [here](https://discuss.hashicorp.com/t/error-using-the-vault-agent-init-to-inject-secrets-from-another-namespace/22681)?
uday avatar
ke flag
Yeah I tried but no luck
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.