Score:0

Ubuntu

NFS server not responding to nfs mount request

Ignacio Más

12/5/22, 2:50 PM

I have an NFS server running on a Ubuntu server on hirsute that stopped answering mount requests after the upgrade and I can't sort out why and a reinstall has not enabled the nfs-server-back. It appears that the auth-rpcgss-module and rpc-svcgssv services aren't running; nfs-config is no longer active but appears to have completed its task and exited. These three had white dots next to them in the output of the list-dependencies command; the others are green. and I've tried reinstalling all packages without success in making NFS-server start. I've tried masking both modules but still the server does not start. I've even put tshark to listen to port 2049 but the server shows no activity, while the client does send the TCP SYN packet to the port without answer. The nfs-server service says that it started and then exited. I'm getting crazy here, anyone that can help in debugging the problem?

This is the output of systemctl nfs-kernel-server status:

$ sudo systemctl status nfs-kernel-server.service 
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: ena>
    Drop-In: /run/systemd/generator/nfs-server.service.d
             └─order-with-mounts.conf
     Active: active (exited) since Thu 2021-08-05 16:33:30 CEST; 8min ago
    Process: 1655760 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 1655761 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SU>
   Main PID: 1655761 (code=exited, status=0/SUCCESS)

Aug 05 16:33:29 calixto systemd[1]: Starting NFS server and services...
Aug 05 16:33:30 calixto systemd[1]: Finished NFS server and services.

Updated with new information: I've tried mounting the NFS share from two different linux client machines without success in any of them. I've sniffed the network with tshark and it seems that the TCP SYN does not arrive to the server, since I cannot see it in the tshark window on the server. I've then tried to mount the shares directly on the server and they mounted without problem so it might seem that there is something with the way the nfs kernel server is opening port 2049. The server has a bridge between a physical network interface and a virtual one serving a KVM vm and it also has many other services running (Apache, a DLNa server and several more) that work without problem! I'm out of hints on how to debug this problem but I'm sure now that it is something related to the way the NFW kernel server listen to port 2049... Any hints?

412

0 + 18

server

nfs

mount

21.04

SEWTGIYWTKHNTDS

12/5/22, 5:36 PM

Is there anything in /etc/exports for the system to export? what does `journalctl -xe` show

Ignacio Más

12/5/22, 5:50 PM

Yes, /etc/exports contains the directories to be exported. Here it comes: `/export 192.168.1.0/24(rw,sync) /export/Backup 192.168.1.3(rw,fsid=0,insecure,no_subtree_check,async) /export/Media 192.168.1.0/24(ro,sync)`

Ignacio Más

12/5/22, 5:55 PM

journalctl does not show anything. That is why I believe it is networking related. A I see the TCP sync coming out of the client but not arriving to the tshark instance on the server. It feels like somehow the TCP session is not being routed, but all the rest of the services work fine, including http, DLNA server, Mosquitto and several others

SEWTGIYWTKHNTDS

12/5/22, 5:59 PM

Is ufw running? anything in the ufw.log relating to nfs or access from the clients?

Ignacio Más

12/5/22, 7:03 PM

No, I don't have ufw installed. No other firewall neither. I see the port with netstat for both tc and tcp6 like this: 'sudo netstat -n -p --tcp --listening | grep 2049 - tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp6 0 0 :::2049 :::* LISTEN -'

Ignacio Más

12/5/22, 7:05 PM

It's baffling that I can mount the NFS exports from the server itself connecting to the external IP '192.168.1.2' but not from any other machine. I simply don't get where the packets are being dropped since all other services work!

SEWTGIYWTKHNTDS

12/5/22, 7:05 PM

If you run `nmap` against the system from one of the client machines which ports are open?

Ignacio Más

12/5/22, 7:08 PM

'nmap -P 192.168.1.2 Starting Nmap 7.80 ( https://nmap.org ) at 2021-08-05 21:07 CEST Nmap scan report for Calixto.MasFunke (192.168.1.2) Host is up (0.00018s latency). Not shown: 987 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 80/tcp open http 111/tcp open rpcbind 139/tcp open netbios-ssn 443/tcp open https 445/tcp open microsoft-ds 2049/tcp open nfs 8080/tcp open http-proxy 8086/tcp open d-s-n 8200/tcp open trivnet1 9090/tcp open zeus-admin 9099/tcp open unknown MAC Address: C8:1F:66:BC:F2:FE (Dell)'

Ignacio Más

12/5/22, 7:09 PM

2049 is open according to Nmap... so there has to be something in the NFS-Kernelserver that is dropping the packets... why? beats me!

SEWTGIYWTKHNTDS

12/5/22, 7:14 PM

Are you expecting all those other ports to be open on the nfs server? I thought the Vm was running those services. what does a scan of the VM IP address show? Does that give any pointers? If nmap sees the ports as open I think the packets are getting through. as you can mount them locally then I am baffled too!

SEWTGIYWTKHNTDS

12/5/22, 7:33 PM

I did come across a problem once where a switch was confused about where it should send packets but I think all packets were going missing, we couldn't ping the server from other servers but you can :(. Your output from all your tests match my servers output and it works. The only difference I can see is in the exports file permissions, I have `/home/git 192.168.57.0/24(rw,sync,no_root_squash,no_all_squash)` perhaps something has changed in the security of the nfs server which is stopping your connections, I would. have thought syslog would show something though

Ignacio Más

12/5/22, 7:53 PM

I'm using the VM only for a home assistant installation. The IP address on the VM (.38) which shares the virtual bridge with the physical interface gives only two open ports (22, ssh and 8082 Home assistant). I have another interface up in the server (.37) on another physical NIC and the Nmap scan on that interface gives the *same* open ports as the bridged interface, including the NFS port. I've tried mounting on that interface but still no answer. Well, thanks for your time in any case! I'll keep on trying things and if I came up with the answer I'll post it as well...

Ignacio Más

12/5/22, 7:54 PM

I'm imclined to think around the scurity options as well... I'll test different ones there

SEWTGIYWTKHNTDS

12/5/22, 8:09 PM

Good luck, hope you sort it soon. I notice that you have two routes to your network on the nfs server, I wonder if that is the issue, the packets might return from a different IP (nfs server) than they were sent to. Why not down the bridge interface on the nfs server and see if that makes any difference.

Ignacio Más

12/6/22, 8:14 AM

Adding more info: I have tested to make a tcptracerote to port 2049 that answers the TCP SYN with a proper [SYN,ACK] and I have even tested telneting from the client machine to port 2049 in the server, which as well responds with a TCP establishment. running tshark in both client and server shows the proper TCP SYN + server SYN/ACK for both the tcptraceroute and telnet case (showing that the IP routing works properly) but in the case of using the mount command in the client I see the TCP SYN from the client but I see *nothing* in the server!!! Can it be a problem with mount on the client?!?!

SEWTGIYWTKHNTDS

12/6/22, 7:04 PM

This doesn't make sense, if you can telnet to the port and the packets arrive at the server but the mount command sends the same syn packets to the same server and they don't arrive. Out of interest, what is the mount command you are using?

Ignacio Más

12/7/22, 8:59 AM

I know! It did not make sense, but now I have managed to isolate the problem. Yes, it was networking related and due to the two interfaces on the same subnet. I put down the second interface of rebooted my switch and now I've mounted the NFS share without problem. Beats me though why telnet to the port worked and the other services, but in any case when I just have the bridged (server + KVM VM) interfaces up in one single physical eth port it works. I guess I have to refresh my knowledge of L2 protocols. :-) there was clearly a L2 loop messing up the NFs protocol. Thanks for all the help!

SEWTGIYWTKHNTDS

12/7/22, 11:53 AM

Great to hear you sorted it

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: NFS server not responding to nfs mount request

TH: เซิร์ฟเวอร์ NFS ไม่ตอบสนองต่อคำขอเมานต์ nfs

RO: Serverul NFS nu răspunde la cererea de montare nfs

RU: Сервер NFS не отвечает на запрос монтирования nfs

VI: Máy chủ NFS không phản hồi yêu cầu gắn kết nfs

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.