Score:0

Kubernetes upgrade from 1.21 to 1.22 caused Prometheus to fail

ir flag

We recently upgraded Kubernetes 1.21 to 1.22 version on aws eks. The upgrade was successful. However, the associated prometheus deployments fails with error

$ kubectl -n monitoring logs prometheus-operator-***
W0109 20:31:28.602872       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"level":"info","msg":"patching webhook configurations 'prometheus-operator-kube-p-admission' mutating=true, validating=true, failurePolicy=Fail","source":"k8s/k8s.go:39","time":"2023-01-09T20:31:28Z"}
{"err":"the server could not find the requested resource","level":"fatal","msg":"failed getting validating webhook","source":"k8s/k8s.go:48","time":"2023-01-09T20:31:28Z"}

In the events prometheus node exporter

 Liveness probe failed: Get "http://*******:9100/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Helm is running on helm3 Any thoughts/directions will be appreciated

tilleyc avatar
us flag
My initial reaction is that you have some Ingresses that are on older API versions. Did you check for deprecations before upgrading?
Score:2
nl flag

The Older Prometheus chart (v11.13.1 from September 2020) is not compatible with Kubernetes v1.22 because the RBAC ClusterRoleBinding API has changed.

Try the Helm chart v15.1.3 on the Kubernetes v1.22 cluster. Please go throuth the link for more information.

As @tilleyc commented, Check Deprecated API Migration Guide on the v1.22 release stopped serving the following deprecated API versions for more details.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.