Score:0

very slow and inconsistent NFS file system response on ubuntu 22.04.2

lv flag
jds

Running Ubuntu 22.04.2 with latest patches. This host is an NFS client running kernel Linux adam408 5.19.0-35-generic #36~22.04.1-Ubuntu

Connectivity is over 25gig Ethernet to Cisco Nexus fabric.

NFS mounts mount fine but performance is very slow and very variable. A user may run

/usr/bin/time -p /usr/bin/ls -l /nfsdata/dir/ | wc -l and get the following

real 9.36 user 0.03 sys 0.57 7127

Real times vary any where from 1.5seconds to 10seconds or higher. The command always completes but takes a very long time. There are only 7127 files in the directory.

Here are the mount options: fsserver:/nfsdata/dir on /nfsdata/dir type nfs4 (rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,nosharecache,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.1.140,local_lock=none,addr=10.1.1.250)

Further context and history:

This is happening across all nodes in a small cluster. All running the same OS kernel and patch level. Prior to an update of the OS (from ubuntu 20.04 to 22.04.2) this was not an issue.

We have tried adjusting various settings in fstab like nordirplus, mountvers, sync as well as many others with no effect. We have also investigated networking both the host and switch interfaces show now errors. Pinging between the Netapp NFS server and the host is sub-milisecond with no packet loss. No other applications seem to be affected. The NetApp is brand new fully patched. We do see this performance problem whether we are accessing a NetApp NFS server or another Ubuntu NFS server. The hosts and the NFS servers are all on the same subnet so there is no routing involved.

We are running out of ideas.

Anyone seen this before?

jds avatar
lv flag
jds
After more testing I took another machine running Ubuntu 22.04.2 with same kernel and patch level and mounted the storage on it over a 1gig adapter and it works fine. no performance issues at all.
jds avatar
lv flag
jds
Otherwise problem still exists on the server hardware.
jds avatar
lv flag
jds
ok this issue us getting weirder. I run my test from my newly built test machine and performance looks good ls -l on the mounted file system takes less than a second. but then i log out and log back in and run the exact same test and it takes 10 to 15 seconds.
karel avatar
sa flag
Does this answer your question? [Surprising slowdown with Apache/NFS on Ubuntu 22.04.2](https://askubuntu.com/questions/1459450/surprising-slowdown-with-apache-nfs-on-ubuntu-22-04-2)
Score:1
yt flag
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.