I am setting up prometheus to scrape kubernetes cluster. I am trying to use "role: node" with kubernetes_sd_config to monitor one of the K8s cluster.
I created certificate ashishcert.pem for user "ashish" and prometheus will use this cert to scrape the cluster. This certificate is signed by cluster CA.
Prometheus.yml
Now when i look back in my prometheus, it says "cannot validate certificate x.x.x because it does not contain any IP SAN's"
result on prometheus side
The port no given in image is for kublet and that means its unable to scrap kublet metrics for all the nodes in cluster. Though i have added all the node names and IPs in SAN of certificate.
i validated my certificate by checking metrics of apisever using my cert and CA cert with below command.
curl -v https://myclustername:6443/metrics --cacert ca.pem --cert ashishcert.pem --key ashishkey.pem
And the above command worked successfully. my cert was accepted by apiserver. However when i tried to curl kublet metrics with path https://myclustername:10250/metrics. it gave me an error saying CA is not trusted. looks like kublet CA is different than apiserver CA.
result while doing curl
I had understanding that my certificate will connect me (prometheus) to apiserver and then its apiserver duty for all further communications like apiserver will use its certificate to get the metrics from kublet. However with results of above commands, looks like mycert is being authenticated directly with kublet also. Please confirm whose certificate will be used for internal communications.
How to scrape all the nodes with role: node without ignoring certificates?
