Join Log in
Score:0
xenrelay avatar Server

Kubernetes managing many distinct UDP servers on GKE

au flag
xenrelay

I'm trying to set up a system that can automatically spin up and down video game servers as docker images. In this case, factoriotools/factorio-docker. Each game is a different, distinct single-pod deployment of that container, and therefore (in the simplified case) needs its own IP address that can listen on a specific UDP port. Load balancers are redundant and irrelevant, and Cloud NAT doesn't appear to allow ingress traffic easily.

There's a couple ways I know of to get this to work, both with pretty major compromises:

  • I can use a NodePort service, and lose control over which port the client needs to connect to. That's an issue because the server registers itself with a server listing.
  • I can use host networking. If my information is correct, that requires privileged containers, which is Definitely Not Good.
  • I could maybe use a UDP load balancer, but even if that exists and works, it's expensive.

There are probably ways to work around the limitations of either approach (for the second, keep the hosts short-lived and keep the firewall strict, and it should be mostly OK?), but I can't help but think there's a better option that I can't find described in the official kubernetes docs. Does traefik have some trick I don't know about? Is there some way to get a variant of MetalLB that can dynamically allocate public IP addresses as I need them?

How do I get each server-container to listen on a different public IP address with a specific UDP port, without making security impossible in the process?

Edits:

  • The port can be configured, so long as I know which port the server needs to run on before the pod starts up.
  • Unless I'm misunderstanding the server docs, the client needs to be able to connect to the same port the server is listening on. The difference k8s draws between the app listening port and the client connecting port is unhelpful in my case.
  • Addressing this question , I don't want a load balancer because it does literally nothing for me. Changing IP addresses do not matter, the system is designed to handle that. If the server goes down for 15s, everyone's going to have to reconnect anyway, and they'll find the new IP address then. The server can't handle multiple pods on the same game, so there will never be more than one replica.
  • I just tried a NodePort service with a randomized public port (hadn't yet seen that I can choose the external port), and I got direct connections to work, but not server listing connections. The server listing process first auto-detects how to connect to the server by having the server send outbound UDP traffic to a given endpoint. So, not only do I need to control what port inbound traffic connects to, I also need to control how outbound traffic is routed, including any NAT that might be used.
266
1 + 4
udp
Dawid Kruk avatar
cn flag
Dawid Kruk
I'd reckon the recommended way to do this would be to create a `Deployment` with a `Service` of type `LoadBalancer` for each game. This would incur costs accordingly to: https://cloud.google.com/vpc/network-pricing. Adding to that, I'm not sure that Traefik is capable of "routing" UDP packets: [source](https://doc.traefik.io/traefik/routing/routers/#configuring-udp-routers). In this setup you have 2 ports that should be configured (`port` (client connects to it) and `targetPort` (app listen)). Can the `port` be changed or should be the same in each game?
0
Reply
xenrelay avatar
au flag
xenrelay
Do those edits help?
0
Reply
Dawid Kruk avatar
cn flag
Dawid Kruk
yes those in fact help. I do not posses much knowledge about `Factorio` but I'd reckon you could set the `Deployment` for each game that would be backed up with `Service` of type `nodePort` [with a specific nodePort port](https://stackoverflow.com/a/60116792/12257134). Acknowledging your other points (omitting the `Service` of type `LoadBalancer`), I cannot tell other solution that would work in that case. Please tell if that would suit you or you have other concerns.
0
Reply
xenrelay avatar
au flag
xenrelay
I'll have to test to see if NodePort works the waythat link describes, but if it does, that would help a lot. The situation with outbound traffic getting re-routed to a different port, and the impact that has on server listing, is probably worthy of a new question, though.
0
Reply
Score:1
Dawid Kruk avatar Server
cn flag
Dawid Kruk

Posting this community wiki answer to set more of a baseline approach to this question rather than to give a definitive solution.

Feel free to edit and expand.

You can expose your applications with Services. There are a few options where each is different in some way from another:

  • ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
  • NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
  • LoadBalancer: Exposes the Service externally using a cloud provider's load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
  • ExternalName: Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up.

-- Kubernetes.io: Docs: Concepts: Services networking: Service: Publishing services service types

The documentation specific to exposing apps on Google Kubernetes Engine can be found here:

Focusing specifically on some of the points included in the question:

I can use a NodePort service, and lose control over which port the client needs to connect to. That's an issue because the server registers itself with a server listing.

You can specify the NodePort port in the Service YAML (like nodePort: 32137 or nodePort: 30911).

You could configure your application to listen on the same port as nodePort:

  • Application is listening on port 30000
  • Service is using a nodePort with port:30000 (client/user should connect to this port) and targetPort:30000. In that case there would be no port changes.

A side note!

By default the nodePort port range is blocked by GCP Firewall. You will need to create a rule (or set of rules) that would allow it.

I can use host networking. If my information is correct, that requires privileged containers, which is Definitely Not Good.

I would be advising against using privileged containers unless a good reason is behind it. Citing the official documentation:

The Privileged policy is purposely-open, and entirely unrestricted. This type of policy is typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users.

-- Kubernetes.io: Docs: Concepts: Security: Pod security standard: Privileged

The port can be configured, so long as I know which port the server needs to run on before the pod starts up.

As you will have a multitude of single Pods (each with a separate Deployment) you could parametrize each of it. What I mean is that you can create a template and modify only the parts of your manifests (like ports, env variables, etc.).

You can pass the environment variable to your Pod so that it can be used as a parameter in your commands. You can also modify the command that the Pod is starting with

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.