Score:0

Server

NFSv4 and kerberos: access denied 50% of the time

francisaugusto

8/2/23, 2:41 PM

We are trying to mount NFSv4 shares on RHEL 8 clients, with kerberos. We have a very similar setup on another environment, and it worked fine. But on this setup, it happens that we get access denied around 50% of the times we try to mount a share:

# failed attempt

bash-4.4$ sudo mount -t nfs -o sec=krb5 server.com:/homes/francis test -vvvv
mount.nfs: timeout set for Sat Apr  2 16:28:32 2022
mount.nfs: trying text-based options 'sec=krb5,vers=4.2,addr=192.168.1.89,clientaddr=192.168.2.29'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'sec=krb5,vers=4,minorversion=1,addr=192.168.1.89,clientaddr=192.168.2.29'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=192.168.1.89,clientaddr=192.168.2.29'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=192.168.1.88,clientaddr=192.168.2.29'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5,addr=192.168.1.89'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.89 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.89 prog 100005 vers 3 prot UDP port 32767
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5,addr=192.168.1.88'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.88 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.88 prog 100005 vers 3 prot UDP port 32767
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting hypatia.uio.no:/uioit-usit-drift-homes/francis

# working attempt two seconds later
bash-4.4$ sudo mount -t nfs -o sec=krb5 server.com:/homes/francis test -vvvv
mount.nfs: timeout set for Sat Apr  2 16:30:09 2022
mount.nfs: trying text-based options 'sec=krb5,vers=4.2,addr=192.168.1.88,clientaddr=192.168.2.29'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'sec=krb5,vers=4,minorversion=1,addr=192.168.1.88,clientaddr=192.168.2.29'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=192.168.1.88,clientaddr=192.168.2.29'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=192.168.1.89,clientaddr=192.168.2.29'

I have checked the logs on the client side, and not much there that points to the cause of the failure to mount. It works one time, and it won't work two seconds later. Or vice-versa.

I thought at first it could be a cross-mount issue, but I also tried with the upper directory of the share, and it was the same problem.

Any hints on what can be the problem?

0 + 0

kerberos

nfs4

rhel8

stark

8/2/23, 2:51 PM

Are the client and server on different subnets? Have you checked router and firewall logs?

Tomek

8/2/23, 3:36 PM

Your `server.com` seems to resolve to two ip addresses: 192.168.1.88 & 192.168.1.89. Do these two addresses have correct reverse DNS mapping, pointing back to `server.com`? Kerberos is pretty picky when it comes to correct DNS configuration.

francisaugusto

8/3/23, 9:09 PM

There's a good point @Tomek. Both addresses have two PTR records each, and that might be the problem. While I do have a setup where the server doesn't have a PTR and it works pretty well, having PTR that doesn't resolve to the host I am using might be making things worse.

francisaugusto

8/3/23, 9:10 PM

@stark yes, different subnets, but network-wise, all is good.

Tomek

8/4/23, 6:55 AM

If you can't resolve DNS issue (which you should) you can try to set `rdns` setting in `krb5.conf` to `false` and see if it helps (it should but sometimes may be overwritten at application level, I believe OpenSSH does that if configured to do so).

francisaugusto

8/4/23, 8:41 AM

@Tomek I tried that with runs (Is there anything that needs to be restarted?), and it didn't work. I am seing if I can remove the PTR records that do not resolve back to the hostname of the principal for nfs mounting to see if that helps.

Tomek

8/4/23, 11:31 AM

Possibly rpc.gssd... But I am not sure.

Score:0

Server

francisaugusto

8/5/23, 7:52 AM

The problem, in my case, was that there were two PTR configured for the server. Even using rdns=false didn't help. When removing the PTR that didn't resolve back to the hostname that match the principal for the server, things worked much better.

Thanks @Tomek for the tip.

0 + 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: NFSv4 and kerberos: access denied 50% of the time

TH: NFSv4 และ kerberos: การเข้าถึงถูกปฏิเสธ 50% ของเวลาทั้งหมด

RO: NFSv4 și kerberos: acces refuzat 50% din timp

RU: NFSv4 и kerberos: доступ запрещен в 50% случаев

VI: NFSv4 và kerberos: truy cập bị từ chối 50% thời gian

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.