Using cluster of eGPUs crashes Ubuntu 21.10

719016

6/10/23, 10:18 AM

We've set up an AMD Ryzen computer with Ubuntu 21.10, and plugged 6 Akitio Duo machines each with 2x NVIDIA 4GB cards, as well as a 13th card directly on the PCIe slot which is a 16GB NVIDIA card (RTX A4000).

We have this rig running 16x threads of Alphafold2 (https://github.com/deepmind/alphafold#running-alphafold) and for the most of it, it can run without issues for a while.

But every once in a while, maybe once every 24hr or so on average, the computer completely locks up. If we only have 4x Alphafold2 running on the 16GB card, the computer is stable for weeks, so the issue seems to be with the jobs on the Akitio eGPU cards.

Is there anywhere that can tell us why it's crashing (the computer is on, but completely unresponsive, only a physical power button reboot does the trick)?

Looking at /var/log/kern.log doesn't seem to show anything indicative of the issue.

EDIT:

Running dmidecode when only the 16GB card plus 2 Akitios are plugged, gives the following:

# dmidecode --type 9 | egrep "Usage|Type|Designation"
        Designation: PCIEX16_1
        Type: x16 PCI Express
        Current Usage: Available
        Designation: PCIEX16_2
        Type: x8 PCI Express
        Current Usage: In Use
        Designation: PCIEX1_1
        Type: x1 PCI Express
        Current Usage: Available

thanks @matigo for the suggestion to look at syslog. In the latest crash, it shows the bit above the '@^' bit, then the hard reboot was at 10:02.

0 + 0

nvidia

drivers

matigo

6/10/23, 10:51 AM

Is there anything in `/var/log/syslog` that points to the issue?

719016

6/10/23, 11:47 AM

Added a screenshot of the before/after of the last crash. The green bit in the middle makes me think it was a serious crash, adding these binary characters between the crash and the reboot. The machine is connected via Ethernet, so NetworkManager shouldn't be having issues with the wireless as the wireless isn't needed to be connected to the network...

Ray

6/10/23, 11:56 AM

By any chance you're using a shared file system such as `OCFS2` or `GFS2`?

719016

6/10/23, 1:40 PM

NFS, yes, mounted like `mount -t nfs url:/this /that`.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Using cluster of eGPUs crashes Ubuntu 21.10

TH: การใช้คลัสเตอร์ของ eGPU ทำให้ Ubuntu 21.10 ขัดข้อง

RO: Utilizarea clusterului de eGPU-uri blochează Ubuntu 21.10

RU: Использование кластера eGPU приводит к сбою Ubuntu 21.10

VI: Sử dụng cụm eGPU làm treo Ubuntu 21.10

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.