Join Log in
Score:0
avatar Server

HA in k8s cluster in case of node failure pointed by A record of domain

cn flag
Jack

I've got a question of "what if" kind

Suppose there is a Kubernetes cluster with 4 nodes and some domain which points to IP of node no 1 and web app using this domain having 1 pod per node. If node 1 will fail then in the current state of my knowledge the app will fail cause A record is pointing to that node which is broken

How it can be solved to maintain HA environment?

51
0 + 0
Score:1
avatar Server
in flag
mdaniel

That is the problem that a Service is designed to solve, and if you are in a cloud environment (or otherwise have an operator that will provision something that looks like a load balancer), then type: LoadBalancer will provision a stable entrypoint from outside the cluster to inside the cluster, and then kubernetes will route around that Node failure.

Under the covers, type: LoadBalancer is just glue between type: NodePort and the load balancer, so even if you don't have a formal load balancer mechanism available to you, using type: NodePort and a copy of haproxy pointed at every Node in your cluster will go a long way toward addressing your risk

0 + 0
cn flag
Jack
So load balancer will be single point of failure in this case too right? in case of it being offline someone will have to change A record to mitigate this failure? or maybe there is an option to make it automatically like "in case of failure node/LB1 point domain to node/LB2" also thank you for answer :)
0
Reply
in flag
mdaniel
Heh, it's turtles all the way down! But seriously, there is _always_ going to be a point of failure but without knowing more about your environment, and the ways you would drive down such a risk _outside_ of kubernetes, it's hard to offer concrete advice for how to drive down the risk _with_ kubernetes. I know that some "bare metal" folks use IPVS+haproxy, others have fancy networking gear that solves the same problem, and others do literally use haproxy just with DNS R-R across the haproxy machines. But to come all the way back to your original question, no, publishing Node IPs is always bad
0
Reply
Score:0
avatar Server
ci flag
JackBack

Thanks mdaniel for clarification!

I've also found useful links to dive deeper

Is it possible to make redundancy on HAProxy server?

How to setup HAProxy with failover?

It's also a good idea to check out topics such as floating ip, keepalived, and if your vendor have api for changing destination of floating ip here on digitalocean you can check how-to https://www.digitalocean.com/community/tutorials/how-to-set-up-highly-available-haproxy-servers-with-keepalived-and-floating-ips-on-ubuntu-14-04

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.