When trying to nfs4 mount a share with sec=krb5 I get "not exported" error message on nfs server

I run two nfs servers with user home directories and other shares on them. The servers offer kerberized nfsv4 mounts as well as old v3 ones.

Until recently I was able to nfs4-mount a krb5-exported share from one server on the second without any problems. If I try now I get an permission denied and in the nfs server log I always see this error shown below. The filesystems with userdata on the nfs servers are located in /export/user1 on host server1.uni-ko.de and /export/user2 on host server2.uni-ko.de (hostnames changed). The path /export/ is used as common nfs4 export root marked by fsid=0.

## run on host server1.uni-ko.de
# mount -t nfs4 -o sec=krb5 server2.uni-ko.de:/user2 /mnt
mount.nfs4: access denied by server while mounting server2.uni-ko.de:/user2

and in the nfs servers log I see this error:

refused mount request from server1.uni-ko.de for /user2 (/): not exported

The same happens if I try to locally mount a share on the same host via nfs4 like this:

## on host server1.uni-ko.de
# mount -t nfs4 -o sec=krb5 server1.uni-ko.de:/user1 /mnt
mount.nfs4: access denied by server while mounting server1.uni-ko.de:/user1

A nfs3 mount (mount server1.uni-ko.de:/export/user1 /mnt) is possible without any problems.

The /etc/exports file on host server2.uni-ko.de looks like this:

/export gss/krb5(rw,fsid=0,nohide,no_subtree_check,no_root_squash,async,crossmnt) \
       gss/krb5i(rw,fsid=0,nohide,no_subtree_check,no_root_squash,async,crossmnt) \
       gss/krb5p(rw,fsid=0,nohide,no_subtree_check,no_root_squash,async,crossmnt)
# NFS V3 exports via NIS netgroups
/export/user2   @nfsv3client(rw,async,no_subtree_check,no_subtree_check,fsid=2000)

exportfs -v shows the shares are (krb5-nfs4) exported:

# exportfs -v
....
/export         gss/krb5(rw,async,wdelay,nohide,crossmnt,no_subtree_check,fsid=0,sec=sys,secure,no_root_squash,no_all_squash)
/export         gss/krb5i(rw,async,wdelay,nohide,crossmnt,no_subtree_check,fsid=0,sec=sys,secure,no_root_squash,no_all_squash)
/export         gss/krb5p(rw,async,wdelay,nohide,crossmnt,no_subtree_check,fsid=0,sec=sys,secure,no_root_squash,no_all_squash)
/export/user2   @nfsv3client(rw,async,wdelay,hide,no_subtree_check,fsid=2000,sec=sys,secure,root_squash,no_all_squash)

Things get even more strange since from a default nfs4 client I am still able to nfs4 mount the given directory without any problems exactly as described above and both systems share the same OS in der very same version and patch level which is SuSE SLES15SP3 with the latest patches installed.

On the nfs servers there is a firewall running opening the statically assigned ports needed for NFS like mountd,statd,lockd as well as ports 111 and 2049 (all opened for tcp and udp). This is what I changed recently. Before this change nfs server ports were not static but set by defaults, but I cannot see how this could lead to the strange mount behavior described. I also disabled the firewall completely and tested again with the same result.

Behind the scenes there is a kerberos server running keeping all the principals needed. For the nfs servers and all nfs4 clients there are "nfs" and "host" principals available and each server and client has a /etc/krb5.keytab containing the exported "nfs" and "host" principal for this host. Of course user principals are also stored in the kerberos server. This all worked flawlessly for years and still does, except for the described problem which is new.

The problem seems to be solved. My way to the solution that I initially had not expected to be one was in two steps:

1. I used the rpcdebug-command to debug nfs rpc calls and found that the error "(/): not exported" was the result of the host trying to NFS4 mount the share which failed (for an unknown reason) and then dropped to a NFS3 mount, which also failed because the mount path given for NFS4 did not contain the path for the NFS4 common root fsid=0 filesystem (/export in my case). So actually nfs4 failed without any visible error and the nfs3 mount failed with a correct error message. So it was clear NFS4 is the culprit and I could forget about the "not exported" error message.

2. After digging around I took a closer look at our kerberos server and noticed it got a security update recently which also resulted in a newer version (1.19.2). I read the MIT docs (https://web.mit.edu/kerberos/krb5-latest/doc/admin/enctypes.html) about steps that should be taken after an version upgrade. One part was the update of several possibly old internal DES- based keys (DES is deprecated and will very soon be dropped). I had one such key on our kerberos server named: kadmin/history . I updated this key as documented using kadmin and after a while I discovered that the NFS4 mount-problem had gone away. However I never saw any error/warning messages on the kerberos server or on the NFS server or client about any kerberos key problem. So I see that nfs4-mounting is working again, but I still doubt a little if it really was only the kerberos des-key that caused this problem ...