Score:0

Ubuntu

Error attaching device to DPDK

Dmitry Dmitriev

5/3/23, 8:36 PM

Error attaching device to DPDK while adding interfaces to OVS bond on Ubuntu.

Ubuntu 20.04.3
DPDK 19.11.10
OVS 2.13.3
NICs Intel x710 (fw 7.10)

Command:

ovs-vsctl add-bond ovsbr0 bond0  ens2f0 ens2f1 ens3f0 ens3f1 \
   -- set Interface ens2f0 type=dpdk  "options:dpdk-devargs=0000:5e:00.0" \
   -- set Interface ens2f1 type=dpdk  "options:dpdk-devargs=0000:5e:00.1" \
   -- set Interface ens3f0 type=dpdk  "options:dpdk-devargs=0000:d8:00.0" \
   -- set Interface ens3f1 type=dpdk  "options:dpdk-devargs=0000:d8:00.1"

Output:

ovs-vsctl: Error detected while setting up 'ens2f0': Error attaching device '0000:5e:00.0' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: Error detected while setting up 'ens2f1': Error attaching device '0000:5e:00.1' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: Error detected while setting up 'ens3f0': Error attaching device '0000:d8:00.0' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: Error detected while setting up 'ens3f1': Error attaching device '0000:d8:00.1' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".

/var/log/openvswitch/ovs-vswitchd.log

2022-01-03T16:00:05.776Z|00083|dpdk|ERR|EAL: Driver cannot attach the device (0000:d8:00.1)
2022-01-03T16:00:05.776Z|00084|dpdk|ERR|EAL: Failed to attach device on primary process
2022-01-03T16:00:05.776Z|00085|netdev_dpdk|WARN|Error attaching device '0000:d8:00.1' to DPDK
2022-01-03T16:00:05.776Z|00086|netdev|WARN|ens3f1: could not set configuration (Invalid argument)
2022-01-03T16:00:05.776Z|00087|dpdk|ERR|Invalid port_id=32

VT-d support is enabled:

cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.4.0-91-generic root=UUID=2849776f-b167-447f-a5d1-ea4b5c831c35 ro iommu=pt intel_iommu=on

I use vfio-pci driver.

DMAR:

$ grep -i dmar charlie-dmesg2.yml | grep -i iommu
[    0.703534] DMAR: IOMMU enabled
[    1.353766] DMAR-IR: IOAPIC id 12 under DRHD base  0xc5ffc000 IOMMU 6
[    1.353767] DMAR-IR: IOAPIC id 11 under DRHD base  0xb87fc000 IOMMU 5
[    1.353768] DMAR-IR: IOAPIC id 10 under DRHD base  0xaaffc000 IOMMU 4
[    1.353770] DMAR-IR: IOAPIC id 18 under DRHD base  0xfbffc000 IOMMU 3
[    1.353771] DMAR-IR: IOAPIC id 17 under DRHD base  0xee7fc000 IOMMU 2
[    1.353773] DMAR-IR: IOAPIC id 16 under DRHD base  0xe0ffc000 IOMMU 1
[    1.353774] DMAR-IR: IOAPIC id 15 under DRHD base  0xd37fc000 IOMMU 0
[    1.353775] DMAR-IR: IOAPIC id 8 under DRHD base  0x9d7fc000 IOMMU 7
[    1.353777] DMAR-IR: IOAPIC id 9 under DRHD base  0x9d7fc000 IOMMU 7

The issue occurs in case of Ubuntu autoinstallation only. Ubuntu installed manually works fine.

0 + 0

networking

drivers

Vipin Varghese

5/4/23, 10:52 AM

can you please share the driver with which you are binding to DPDK. Is it uio_pci_generic, igb_uio or vfio-pci? If it is vfio-pci can you please check DMAR for iommu from dmesg?

Dmitry Dmitriev

5/5/23, 4:24 PM

I use vfio-pci: `$ grep -i dmar charlie-dmesg2.yml | grep -i iommu [ 0.703534] DMAR: IOMMU enabled [ 1.353766] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6 [ 1.353767] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5 [ 1.353768] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4 [ 1.353770] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3 [ 1.353771] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2 [ 1.353773] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1 [...]`

Vipin Varghese

5/6/23, 4:17 AM

thanks for the response, as per the dmesg DMAR logs IOMMU is enabled on your machine. can you check in dmesg once the bind fails if its log ` Cannot bind to driver vfio-pci`? If yes, then vfio-pci bind with default iommu is failing. Please try `echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode` and share.

Dmitry Dmitriev

5/11/23, 4:25 PM

@VipinVarghese thank you for helping me. The dmesg output doesn't contain any messages about vfio-pci. Just in case, I tried the command you sent. No luck.

Vipin Varghese

5/12/23, 3:42 AM

very interesting issue, are you using Intel x86 or and x86 processor? If it is AMD `amd_iommu=pt` is to be used. But since you mentioned manual installation works, I would like to look more into this behaviour will it be possible for a debug?

Dmitry Dmitriev

5/15/23, 4:14 AM

My colleague suggested me to install the package **libdpdk-dev**. It pulls the whole bunch of dependencies, but finally solves the issue. Good luck!

Vipin Varghese

5/15/23, 12:10 PM

as per the officeal package information https://packages.debian.org/sid/libdpdk-dev, it implies these are `DPDK libraries compiled with SSE as minimum`. Hence I am unable to understand how manual DPDK 19.11.10 is used to solve the issue. SOmething is not rigyt.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Error attaching device to DPDK

TH: เกิดข้อผิดพลาดในการเชื่อมต่ออุปกรณ์กับ DPDK

RO: Eroare la atașarea dispozitivului la DPDK

RU: Ошибка присоединения устройства к DPDK

VI: Lỗi khi gắn thiết bị vào DPDK

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.