We need to enable JWT auth in vault which is hosted within our EKS cluster in preparation for using K8s 1.24 OIDC and testing token renewal with Vault. I'm following documentation from a few places:
https://banzaicloud.com/blog/kubernetes-oidc/
https://www.vaultproject.io/docs/auth/jwt/oidc-providers/kubernetes
https://www.vaultproject.io/docs/auth/kubernetes#discovering-the-service-account-issuer
https://learn.hashicorp.com/tutorials/vault/agent-kubernetes?in=vault/auth-methods
The configuration seems pretty straightforward. Where I'm lost is when it's time to write the JWT configuration to vault, the configuration fails with:
vault write -tls-skip-verify auth/jwt/config oidc_discovery_url=https://oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxx oidc_discovery_ca_pem=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt bound_issuer=https://oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxx
...
...
...
* error checking oidc discovery URL: error creating provider with given values: Get "https://vault.vault:8200/.well-known/openid-configuration": x509: certificate signed by unknown authority
Grabbing the CA:
at 22:37:06 ❯ \kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode | openssl x509 -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Oct 15 05:07:03 2022 GMT
Not After : Oct 12 05:07:03 2032 GMT
Subject: CN = kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
The CA used there is the CA which is provided everywhere for the cluster. It's in the kubeconfig, in the issuer for the cluster, and mounted with the serviceaccount data within pods as is what I used in the above command. There are no other CAs of which I'm aware. Moreover, using -tls-skip-verify
is not honored, so that confuses me a little.
Any pointers on what's going on here would be greatly appreciated.