Automating CPU pinning for offload worker queue in HWOL/Conntrack offload environment

I am currently testing for the introduction of HWOL and conntrack offload. In this process, I am trying to apply CPU pinning to softirq and offload worker queue.

For CPU pinning of the offload worker queue, I am planning to apply it by entering the CPU bitmask in the /sys/devices/virtual/workqueue/nf_ft_offload_*/cpumask file.

I have verified up to the part where the offload workqueue task is properly assigned with the CPU bitmask specified in this file.

root@Qaamdhost02:/opt/script# ll /sys/devices/virtual/workqueue/nf_ft_offload_*/cpumask
drwxr-xr-x 3 root root 0 Jul  4 15:53 /sys/devices/virtual/workqueue/nf_ft_offload_add/cpumask
drwxr-xr-x 3 root root 0 Jul  4 15:53 /sys/devices/virtual/workqueue/nf_ft_offload_del/cpumask
drwxr-xr-x 3 root root 0 Jul  4 15:53 /sys/devices/virtual/workqueue/nf_ft_offload_stats/cpumask

I tried to automate the above setting, but the /sys/devices/virtual/workqueue/nf_ft_offload_* type directory is only created when the conntrack offload task occurs, so it is difficult to automate before or after the VM is deployed.

I am curious if there is another way for CPU pinning of the offload worker queue, or if there is a way to create this directory immediately after the VM is deployed.

Hardware:

  Server: Dell R7615
  CPU: AMD Epyc 9654P
  Memory: 384GB
  NUMA: 1
  NIC: Connect-X 6LX

Software Versions:

  OS: Ubuntu 22.04
  Kernel: 5.15
  Openstack Version: Yoga
  OVN: 22.03
  OVS: 2.17.5
  MLNX OFED Driver: 5.8-2.0.3
  Firmware: 26.35.1012 (DEL0000000031)

According to the response from the Nvidia forum, it was confirmed that they do not support an API for querying tasks when registering a Linux Worker Queue for HWOL /w Conntrack Offload work.

From our own tests, it appears that the mlx5_core Driver registers a Linux Worker Queue task when a VM is created.

Considering that the Representation Port connected to the VM is activated when the VM is created, we conducted a test using the carrier event of networkd-dispatcher as follows:

cat /etc/networkd-dispatcher/carrier.d/03-offload_pinned
#!/usr/bin/env bash
echo 0000000e,0000000e > /sys/devices/virtual/workqueue/nf_ft_offload_add/cpumask
echo 0000000e,0000000e > /sys/devices/virtual/workqueue/nf_ft_offload_del/cpumask
echo 0000000e,0000000e > /sys/devices/virtual/workqueue/nf_ft_offload_stats/cpumask

We confirmed that it works normally when a VM is created.

Could you please let us know if there are any potential issues with this configuration or if there is a better approach?