Score:2

How does RDMA transfer between two servers without the CPU?

mx flag

I've been reading about RoCE and it talks about transferring data zero copy between the memory in one server directly to the memory of an application in another server. Most articles make a point to say this is done without involvement from the CPU. I don't understand how this is accomplished though. How does the data get from the adapter to the RAM without any processing from the CPU? How does it know where the application's data is in RAM?

djdomi avatar
za flag
well providibg learning material or request are off-topic, start reading Wikipedia https://en.m.wikipedia.org/wiki/Remote_direct_memory_access whixh explains at most all sub questions you currently have
Score:3
cn flag

I think you misunderstand "without CPU". It does not mean the CPU is not involved at all - it just means that the CPU is not doing the copying. It does - via drivers - the setup of the copy operation and then the hardware does the actual processing.

Your link says so VERY explicitly - just read it:

RDMA supports zero-copy networking by enabling the network adapter to transfer data directly from the wire

That is the copy operation, not the setup. Someone has to tell the network adapter.

Understand that modern networks where this really does shine are 100+ gigabit - 10 gigabyte+ - per second, often multiple lines combined, with high end these days being 400gigabit.

That is a lot - for the CPU to read from RAM, loop over it then send it to the Adapter, when you could just tell the adapter which memory to read and send.

But again, "no involvement" is not "no CPU at all", it is "CPU is not copying the data". It still does all the control.

br flag
You're 100% correct (as usual Tom ;) ) - I've only ever used it on Infiniband as RDMA, rather than ROCE, but it is freaky how little CPU impact you get, there is some as you say but it's a wonderful technology - and of course allows for shared memory blocks between many servers - useful for things like cache-coherency. Reliable too!
Grant Curell avatar
mx flag
So in this case then the CPU would still provide a read/write address to the NIC, the NIC would then transfer the data off the wire to the address specified in memory directly?
cn flag
Yeah. Note that depending on how the hardware is set up - devices may map buffer space into memory space. YOu get the idea? Like you could have a read buffer ON THE NIC that then the storage controller directly reads. There is a LOT of stuff in high end - i.e. RAID 5/6. There are now Network cards that can do the calculation. Why? Because RAID 6 may men discs on multiple servers.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.