Join Log in
Score:6
Levi Hibbitts avatar Server

Connecting hundreds of servers via the local network (LAN)

in flag
Levi Hibbitts

I'm caching lots of data in memory to a local Redis server. The server will need to connect to hundreds of other servers in the same data center (as they all need to access the same information hosted by the Redis server).

In order to minimize latency and maximize throughput I plan on connecting all of the servers to the Redis server with a 40 Gigabit Ethernet (40 Gbit/s).

I understand how to connect a couple of servers to one server, but how would I go about connecting hundreds of servers to a single server? I mean, I don't see enough ports to connect so many servers...

Is there a device that will allow me to connect all of these servers together?

2823
0 + 0
in flag
NiKiZe
Do you really need that bandwidth? You will have to create a star network just like with any other Ethernet setup. bond/trunk multiple ports for higher bandwidth and single link failure, consider multipath, as well as multiple synced redis instances.
0
Reply
mfinni avatar
cn flag
mfinni
As kindly as possible : please hire a sysadmin for whatever you're doing. Hire a good one. Spend a lot of money on them.
43
Reply
sa flag
user253751
You have tested this design on plain old 1Gb ethernet and you found that Ethernet bandwidth was a problem, right? Or you calculated it, at least?
4
Reply
sa flag
user253751
Redis is apparently single-threaded. Can one thread push 40Gb of bandwidth? I really doubt it - you'll probably be slowed down by Redis. You could use more than one copy of Redis, but if you can use 5 copies of Redis on one server, you can also use one copy on 5 servers (that you already have) and save money by not buying the one big server just for Redis.
9
Reply
David Browne - Microsoft avatar
in flag
David Browne - Microsoft
Typically each rack has an Ethernet Switch, and each of these uplinks to a main Ethernet Switch. https://www.cisco.com/c/en/us/products/switches/what-is-an-ethernet-switch.html#~q-a
1
Reply
sa flag
user253751
It's pretty strange to encounter a programmer who's never heard of an Ethernet switch, but [you're one of today's lucky 10000](https://xkcd.com/1053/)
16
Reply
Criggie avatar
in flag
Criggie
Consider your redundancy setup too - one single redis server is a SPOF and if it is production-impacting then an outage will be bad.
2
Reply
cn flag
RedBaron
"Hundreds of servers in the data-center" - Is the data-center already setup? If yes, your server can just be connected to a switch in the existing network of the data-center. If not, then the question should be "How do I setup a data-center with 100s of servers".
5
Reply
cn flag
thkala
So... machines fail. A lot. Have you thought what happens if your single cache server fails? Or needs to be taken down for maintenance? Perhaps some sort of Redis cluster would be a better fit?
1
Reply
in flag
marsh-wiggle
If anyone is looking for a question that is getting an unjustified amount of positive attention, here it is.
0
Reply
Bib avatar
cn flag
Bib
If these are all the same o/s, and one o/s is compromised, then all the others will be too in very short order.
0
Reply
ve flag
Mikko Rantalainen
If you truly don't understand or wasn't previously aware what a network switch does, I would recommend researching network hardware a lot more before investing on actual hardware. 40 Gbps connections are pretty expensive and you shoudn't get all that hardware just for fun. Even 2x10 Gbps in 802.3ad (LACP) configuration with redundant switches gets you plenty of bandwidth and the latency for each server often starts to be bigger bottleneck than the network bandwidth. And you don't lose the whole network when a single switch fails. When you have more than 3 servers, you should make it redundant.
0
Reply
Score:14
joeqwerty avatar Server
cv flag
joeqwerty

I understand how to connect a couple of servers to one server but how would I go about connecting hundreds of servers to a single server? I mean, I don't see enough ports to connect so many servers.

Well... you would need to connect all of the servers to an appropriately sized network switch. They've been in use for decades. The first commercial multiport Ethernet network switch was introduced in 1989.

0 + 0
Score:9
Zac67 avatar Server
ru flag
Zac67

I'm caching lots of data in memory to a local redis server. The server will need to connect to hundreds of other servers in the same data center (as they all need to access the same information hosted by the redis server).

The amount of data you store/cache is rather irrelevant. Relevant is the speed at which you need to transmit data to or from storage.

In order to minimize latency and maximize throughput I plan on connecting all of the servers to redis server with a 40 Gigabit Ethernet (40GbE).

For a large number of high-volume network ports you need a good infrastructure design. You not only require a huge bandwidth inside the redis server - network, storage and processing - but also the means to distribute that bandwidth.

Depending on the exact size, a large chassis switch (up to 800 ports or so) or a hierarchical tree is required. This paper from Cisco should provide a good starting point. A collapsed-core design is likely sufficient for your size.

0 + 0
cn flag
Jacob Krall
“hundreds of servers” talking to a single-threaded Redis instance don’t need a developer to buy a million dollar switch. They need to use that money to hire someone who has *heard* of a switch, and then buy a couple of 48-port unmanaged Gigabit switches.
8
Reply
Zac67 avatar
ru flag
Zac67
@JacobKrall We don't know how well that Redis server scales or whether it is *single-threaded* - I was trying to point out the necessity to evaluate the *network, storage and processing* requirements.
0
Reply
Score:2
avatar Server
se flag
An-dir

Well, it depends on your needs.

You may ask your distributor where you want to buy that hardware. They want to help you to get that deal.

One example:

If the hundreds of servers have only gigabit you might use up to 10x HPE 2930M stacked with stacking modules and stacking cables. Then add a 40 Gbit/s module in at least two of them for redundancy and connect your server with both modules via DAC cables.

The 2930M has up to 48 ports, so 480 servers + the one 40G server. Depending in your needs select different switches.

If you don't need the flexibility, but more throughput you should go with a modular switch.

You can find help on several vendors sites, but here I give you HPE’s solution for finding a switch if you want to do it all alone:

HPE Networking Switch Selector

Again: ask someone who sells that solution to you if he/she can help with the order. Otherwise, you might get in trouble when trying to put everything together and you notice that there are different power supplies, modules, cables that fit into different models. Some 2930Ms, for example, need more power than other 2930M models.

0 + 0
fraxinus avatar
ng flag
fraxinus
Asking a salesperson for a complex technical solution is a recipe for a disaster.
6
Reply
se flag
An-dir
For that size of course not a salesperson. The Distributors i worked with had technicans/architects to help in cases like that
0
Reply
cn flag
thkala
Asking anyone with a vested interest in upselling, whether that's a salesperson or the distributor's technical advisor, is still a recipe for disaster :-)
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.