I have a K8s cluster running across 2 VM hosts, with the 6 k8s nodes and 3 masters split across both. We recently had an issue where one of the servers went down leaving me with only one master and 3 nodes. All my services are still running but I can't access the API. Looking at the remaining master the kube API server container does not come up as it seems it's unable to register itself.
Api Server container logs are as follows
I0831 09:25:10.464033 1 server.go:563] external host was not specified, using 192.168.50.51
I0831 09:25:10.464668 1 server.go:161] Version: v1.25.4
I0831 09:25:10.464708 1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0831 09:25:11.015314 1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0831 09:25:11.016581 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0831 09:25:11.016605 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0831 09:25:11.017954 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0831 09:25:11.017970 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
^[[A
W0831 09:25:27.246979 1 logging.go:59] [core] [Channel #3 SubChannel #5] grpc: addrConn.createTransport failed to connect to {
"Addr": "127.0.0.1:2379",
"ServerName": "127.0.0.1",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:47046->127.0.0.1:2379: read: connection reset by peer"
W0831 09:25:27.246980 1 logging.go:59] [core] [Channel #4 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
"Addr": "127.0.0.1:2379",
"ServerName": "127.0.0.1",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:47040->127.0.0.1:2379: read: connection reset by peer"
W0831 09:25:27.247066 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "127.0.0.1:2379",
"ServerName": "127.0.0.1",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:47036->127.0.0.1:2379: read: connection reset by peer"
E0831 09:25:31.020654 1 run.go:74] "command failed" err="context deadline exceeded"