Join Log in
Score:1
avatar Server

Infiniband OpenSM N-to-N port routing configuration

tc flag
Hugo Maxwell

I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch.

My RDMA application runs as one process per NUMA node and binds to the local NIC to avoid cross CPU traffic. Each node/process needs to connect to every other node using RC mode.

The problem I ran into is, it appears the default OpenSM routing forces me to use a certain NIC to reach a certain target node. So I would have to use both NICs from both NUMA nodes to reach all other nodes. That means I would need two PDs also, having to register all the memory twice.

Is there any way to allow a single NIC to be able to connect to any other NIC/port on the network?

Essentially I would like to make OpenSM think that each NIC is on it's own server, ie. pretending that no QPI traffic is possible.

See: https://docs.mellanox.com/display/MLNXOFEDv461000/OpenSM

Once MinHop matrices exist, each switch is visited and for each target LID a decision is made as to what port should be used to get to that LID.

Relevant code: https://github.com/linux-rdma/opensm/blob/844ab3b7edaad983449b5d3a4a773088b8daa299/opensm/osm_ucast_mgr.c#L201

54
0 + 0
hpc
Score:1
avatar Server
tc flag
Hugo Maxwell

https://community.mellanox.com/s/question/0D51T00006RVtlU/rdmacm-connection-setup-issues

For the record, the issue has been largely resolved by ensuring the ibacm service (Infiniband Assistant Communication Manager) was running on all servers.

Running sudo ibacm on all servers solved the issue, don't ask me why...

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.