Permission denied for user who owns the file and the directory

User www-data, owns the "/shared/data" directory, and full permissions are graned to the "/shared" directory.

root@cloud:/shared# ls -al
total 28
drwxrwxrwx  4 root     root     4096 Sep 16 19:06 .
drwxr-xr-x 21 root     root     4096 Sep 16 00:44 ..
drwxrwx--- 21 www-data www-data 4096 Nov  7 16:02 data
drwxrwxrwx  2 root     root     4096 Sep 16 00:22 @Recycle

root@cloud:/# ls -al /
...
drwxrwxrwx   4 root root       4096 Sep 16 19:06 shared
...

There are no files owned by any other user in this directory.

for i in $(find /shared/data); do stat -c "%U %G" ''"$i"'' >> /tmp/output.txt; done
cat /tmp/output.txt | sort -u
www-data www-data

Selinux is set to permissive.

However, when I run:

root@cloud:/shared# sudo -u www-data ls -al /shared/
ls: cannot open directory '/shared/': Permission denied
root@cloud:/shared# sudo -u www-data ls -al /shared/data/
ls: cannot access '/shared/data/': Permission denied

The location "/shared" is mounted VIA NFS using the following entry in "/etc/fstab".

server.fully.qualified.domain:/shared/data /shared/ nfs4 _netdev,nofail,rw  0   0

What in the world is going on here? This was actually working a couple of weeks ago and I didn't change anything of import.

Audit log shows nothing of interest, syslog shows nothing of interest, dmesg shows nothing of interest.

Pathllld shows:

root@cloud:/# sudo -u www-data /pathlld/pathlld /shared/data/
drwxr-xr-x 22 root root 4096 Nov  7 23:23 /
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime)
drwxrwxrwx 4 root root 4096 Sep 16 19:06 /shared
server.fully.qualified.domain:/shared/data on /shared type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.4,local_lock=none,addr=192.168.1.2,_netdev)
/bin/ls: cannot access '/shared/data': Permission denied

root@cloud:/# sudo -u www-data /pathlld/pathlld /shared
drwxr-xr-x 22 root root 4096 Nov  7 23:23 /
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime)
drwxrwxrwx 4 root root 4096 Sep 16 19:06 /shared
server.fully.qualified.domain:/shared/data on /shared type nfs4      (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.4,local_lock=none,addr=192.168.1.2,_netdev)

The answer to this is that there is a security function in NFS where the user permissions are checked on the NFS server as well as the client. Adding a www-data user to the server and making the user id's and the group id's match between the two servers makes it work again. Someone actually deleted the www-data user from the NFS server which made it all stop very suddenly when it had been working.