Running Ubuntu 22.04.2 with latest patches. This host is an NFS client running kernel
Linux adam408 5.19.0-35-generic #36~22.04.1-Ubuntu
Connectivity is over 25gig Ethernet to Cisco Nexus fabric.
NFS mounts mount fine but performance is very slow and very variable. A user may run
/usr/bin/time -p /usr/bin/ls -l /nfsdata/dir/ | wc -l
and get the following
real 9.36
user 0.03
sys 0.57
7127
Real times vary any where from 1.5seconds to 10seconds or higher. The command always completes but takes a very long time. There are only 7127 files in the directory.
Here are the mount options:
fsserver:/nfsdata/dir on /nfsdata/dir type nfs4 (rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,nosharecache,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.1.140,local_lock=none,addr=10.1.1.250)
Further context and history:
This is happening across all nodes in a small cluster. All running the same OS kernel and patch level. Prior to an update of the OS (from ubuntu 20.04 to 22.04.2) this was not an issue.
We have tried adjusting various settings in fstab like nordirplus, mountvers, sync as well as many others with no effect. We have also investigated networking both the host and switch interfaces show now errors. Pinging between the Netapp NFS server and the host is sub-milisecond with no packet loss. No other applications seem to be affected. The NetApp is brand new fully patched. We do see this performance problem whether we are accessing a NetApp NFS server or another Ubuntu NFS server. The hosts and the NFS servers are all on the same subnet so there is no routing involved.
We are running out of ideas.
Anyone seen this before?
