Able to get Kerberised NFSv4 export mounted once, not subsequently. 'Access denied by server while mounting'; 'Additional pre-authentication required'

Able to get Kerberised NFSv4 export mounted once, not subsequently. 'Access denied by server while mounting'; 'Additional pre-authentication required'

Hi,

I'm new to Kerberos, but have some experience with NFS without security.

I'm trying to set up NFS with krb5p security on two hosts running Ubuntu 22.04: the server — enya.colonelpanic.local — is running a fresh install of Ubuntu Server 22.04 on bare metal; the client — imogen.colonelpanic.local — is a VM which was originally running 18.04, but that kept falling back to NFSv3 when v4 was specified in the mount command, so I thought I'd upgrade to see if I'd have any more luck with them both running 22.04.

I've mainly been referring to http://techpubs.spinlocksolutions.com/dklar/kerberos.html and https://wiki.debian.org/NFS/Kerberos I'm not using GSSProxy as outlined in the Debian Wiki article.

I was running the command outlined in the Spinlock Solutions article (sudo tail -n0 -F /var/log/{*log,dmesg,messages,kerberos/{krb5kdc,kadmin,krb5lib}.log}) to monitor log files and wondering why it wasn't producing any output for the Kerberos log files; then I realised that /var/log was being reported as a read-only filesystem, so based on https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1874915 I added /var/log to ReadWriteDirectories in /lib/systemd/system/krb5-kdc.service and /lib/systemd/system/krb5-admin-server.service.

At one point when receiving mount errors, I realised there were missing principals for enya.colonelpanic.local (although I set the Kerberos server for the realm to be krb.colonelpanic.local), so I added those and at one point was eventually able to mount enya.colonelpanic.local:/mnt/foo successfully on imogen, however after rebooting imogen I have been unable to mount, with a 'permission denied by server' error. I also realised at one point that enya's clock (which wasn't using NTP) was about 10 minutes behind imogen's (which was); both are now using NTP.

The first mount attempt following a reboot of imogen results in the following on enya:

==> /var/log/kerberos/krb5kdc.log <==
Jul 09 12:40:08 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: NEEDED_PREAUTH: nfs/[email protected] for krbtgt/[email protected], Additional pre-authentication required
Jul 09 12:40:08 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: ISSUE: authtime 1688902808, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, nfs/[email protected] for krbtgt/[email protected]
Jul 09 12:40:08 enya krb5kdc[71839](info): TGS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: ISSUE: authtime 1688902808, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, nfs/[email protected] for nfs/[email protected]

This is the case whether I issue kinit to obtain a ticket as my user (ansel) or not; I've also tried sudo -s-ing to root, then issuing kinit ansel and mounting, but that has the same result. Subsequent mount attempts result in no additional output in /var/log/kerberos/krb5kdc.log.

The Debian Wiki article states that 'Additional pre-authentication required' errors are due to incorrect keytabs, but the keytabs are correct as far as I know:

ansel@enya:~$ sudo klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- ---------------------------------------------
   8 host/[email protected]
   8 host/[email protected]
   8 nfs/[email protected]
   8 nfs/[email protected]
   8 host/[email protected]
   8 host/[email protected]
   8 nfs/[email protected]
   8 nfs/[email protected]
ansel@enya:~$ 

ansel@imogen:~$ sudo klist -k
[sudo] password for ansel: 
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- ---------------------------------------------
  13 host/[email protected]
  13 host/[email protected]
  16 nfs/[email protected]
  16 nfs/[email protected]
ansel@imogen:~$ 

I've tried deleting and recreating the keytabs, as well as deleting the principals for my user (ansel and ansel/admin), as per https://serverfault.com/a/959566/371237 with no change.

I also found this https://askubuntu.com/questions/1457852/kerberized-nfs-mounts-stopped-working-with-ubuntu-21-10-still-in-22-10 regarding a mising module to do with GSS in Ubuntu 22.04, so made sure rpcsec_gss_krb5 was loaded on both machines with modprobe.

Other relevant config files and command output:

/etc/hosts on enya (server) and imogen (client) (my DNS setup is flaky, so I've specified hostnames and FQDNs here):

ansel@enya:~$ cat /etc/hosts
127.0.0.1 localhost
## Changed for Kerberos setup.
#127.0.1.1 enya
192.168.2.3 enya.colonelpanic.local enya krb.colonelpanic.local krb

192.168.2.124 imogen.colonelpanic.local imogen

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ansel@enya:~$ 

ansel@imogen:~$ cat /etc/hosts
127.0.0.1       localhost
##127.0.1.1       imogen
192.168.2.124 imogen.colonelpanic.local imogen

## Changed for Kerberos setup.
#127.0.1.1 enya
192.168.2.3 enya.colonelpanic.local enya krb.colonelpanic.local krb

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ansel@imogen:~$ 



ansel@enya:~$ cat /etc/krb5.conf
[libdefaults]                                                                                                          
        default_realm = COLONELPANIC.LOCAL                                                                             
                                                                                                                       
# The following krb5.conf variables are only for MIT Kerberos.                                                         
        kdc_timesync = 1                                                                                               
        ccache_type = 4                                                                                                
        forwardable = true                                                                                             
        proxiable = true                                                                                               
                                                                                                                       
# The following encryption type specification will be used by MIT Kerberos                                             
# if uncommented.  In general, the defaults in the MIT Kerberos code are                                               
# correct and overriding these specifications only serves to disable new                                               
# encryption types as they are added, creating interoperability problems.                                              
#                                                                                                                      
# The only time when you might need to uncomment these lines and change                                                
# the enctypes is if you have local software that will break on ticket                                                 
# caches containing ticket encryption types it doesn't know about (such as                                             
# old versions of Sun Java).

#       default_tgs_enctypes = des3-hmac-sha1
#       default_tkt_enctypes = des3-hmac-sha1
#       permitted_enctypes = des3-hmac-sha1

# The following libdefaults parameters are only for Heimdal Kerberos.
        fcc-mit-ticketflags = true

[realms]
        COLONELPANIC.LOCAL = {
                kdc = krb.colonelpanic.local
                admin_server = krb.colonelpanic.local
        }
        ATHENA.MIT.EDU = {
                kdc = kerberos.mit.edu
                kdc = kerberos-1.mit.edu
                kdc = kerberos-2.mit.edu:88
                admin_server = kerberos.mit.edu
                default_domain = mit.edu
        }
        ZONE.MIT.EDU = {
                kdc = casio.mit.edu
                kdc = seiko.mit.edu
                admin_server = casio.mit.edu
        }
        CSAIL.MIT.EDU = {
                admin_server = kerberos.csail.mit.edu
                default_domain = csail.mit.edu
        }
        IHTFP.ORG = {
                kdc = kerberos.ihtfp.org
                admin_server = kerberos.ihtfp.org
        }
        1TS.ORG = {
                kdc = kerberos.1ts.org
                admin_server = kerberos.1ts.org
        }
        ANDREW.CMU.EDU = {
                admin_server = kerberos.andrew.cmu.edu
                default_domain = andrew.cmu.edu
        }
        CS.CMU.EDU = {
                kdc = kerberos-1.srv.cs.cmu.edu
                kdc = kerberos-2.srv.cs.cmu.edu
                kdc = kerberos-3.srv.cs.cmu.edu
                admin_server = kerberos.cs.cmu.edu
        }
        DEMENTIA.ORG = {
                kdc = kerberos.dementix.org
                kdc = kerberos2.dementix.org
                admin_server = kerberos.dementix.org
        }
        stanford.edu = {
                kdc = krb5auth1.stanford.edu
                kdc = krb5auth2.stanford.edu
                kdc = krb5auth3.stanford.edu
                master_kdc = krb5auth1.stanford.edu
                admin_server = krb5-admin.stanford.edu
                default_domain = stanford.edu
        }
        UTORONTO.CA = {
                kdc = kerberos1.utoronto.ca
                kdc = kerberos2.utoronto.ca
                kdc = kerberos3.utoronto.ca
                admin_server = kerberos1.utoronto.ca
                default_domain = utoronto.ca
        }

[domain_realm]
        .mit.edu = ATHENA.MIT.EDU
        mit.edu = ATHENA.MIT.EDU
        .media.mit.edu = MEDIA-LAB.MIT.EDU
        media.mit.edu = MEDIA-LAB.MIT.EDU
        .csail.mit.edu = CSAIL.MIT.EDU
        csail.mit.edu = CSAIL.MIT.EDU
        .whoi.edu = ATHENA.MIT.EDU
        whoi.edu = ATHENA.MIT.EDU
        .stanford.edu = stanford.edu
        .slac.stanford.edu = SLAC.STANFORD.EDU
        .toronto.edu = UTORONTO.CA
        .utoronto.ca = UTORONTO.CA

        .colonelpanic.local = COLONELPANIC.LOCAL
        colonelpanic.local = COLONELPANIC.LOCAL
         
[logging]
        kdc = FILE:/var/log/kerberos/krb5kdc.log
        admin_server = FILE:/var/log/kerberos/kadmin.log
        default = FILE:/var/log/kerberos/krb5lib.log
ansel@enya:~$ 



ansel@enya:~$ sudo kadmin.local -q "listprincs"
Authenticating as principal root/[email protected] with password.
K/[email protected]
ansel/[email protected]
[email protected]
host/[email protected]
host/[email protected]
host/[email protected]
kadmin/[email protected]
kadmin/[email protected]
kadmin/[email protected]
krbtgt/[email protected]
nfs/[email protected]
nfs/[email protected]
nfs/[email protected]
ansel@enya:~$ 

One thing I noticed is that I don't have a principal for kadmin/[email protected] or kadmin/[email protected].

ansel@enya:~$ cat /etc/default/nfs-kernel-server
# Number of servers to start up                                                                                        
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gids" 

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".

## CHANGED FROM EMPTY. (https://wiki.debian.org/NFS/Kerberos)
NEED_SVCGSSD="yes"

# Options for rpc.svcgssd.
## CHANGED (https://wiki.debian.org/NFS/Kerberos)
RPCSVCGSSDOPTS="-vvv"
ansel@enya:~$ 

ansel@enya:~$ cat /etc/default/nfs-common
# If you do not set values for the NEED_ options, they will be attempted                                               
# autodetected; this should be sufficient for most people. Valid alternatives                                          
# for the NEED_ options are "yes" and "no".

# Do you want to start the statd daemon? It is not needed for NFSv4.
## CHANGED FROM EMPTY.
NEED_STATD=yes

# Options for rpc.statd.
#   Should rpc.statd listen on a specific port? This is especially useful
#   when you have a port-based firewall. To use a fixed port, set this
#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".
#   For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS
STATDOPTS=

# Do you want to start the idmapd daemon? It is only needed for NFSv4.
## CHANGED FROM NOTHING.
NEED_IDMAPD=yes

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
## CHANGED FROM NOTHING.
NEED_GSSD=yes
ansel@enya:~$ 

ansel@imogen:~$ cat /etc/default/nfs-common                                                                     [2/231]
# If you do not set values for the NEED_ options, they will be attempted                                               
# autodetected; this should be sufficient for most people. Valid alternatives                                          
# for the NEED_ options are "yes" and "no".

# Do you want to start the statd daemon? It is not needed for NFSv4.
NEED_STATD=

# Options for rpc.statd.
#   Should rpc.statd listen on a specific port? This is especially useful
#   when you have a port-based firewall. To use a fixed port, set this
#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".
#   For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS
STATDOPTS=

# Do you want to start the idmapd daemon? It is only needed for NFSv4.
## CHANGED FROM NOTHING.
NEED_IDMAPD=yes

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=yes

## Added (https://wiki.debian.org/NFS/Kerberos)
RPCGSSDOPTS="-vvv"
ansel@imogen:~$ 

ansel@enya:~$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#

#/srv/nfs/mnt/foo       *(ro,sync,no_root_squash)
##/srv/nfs/mnt/bar       *(rw,sync,no_root_squash)
##/srv/nfs/mnt/baz       *(rw,sync,no_root_squash)

##/mnt/foo   *(rw,sync,no_subtree_check,sec=krb5)
/mnt/foo   gss/krb5p(rw,sync,no_subtree_check)
##/mnt/foo     *(rw,sync,no_root_squash)
ansel@enya:~$ 

An attempt at mounting:

ansel@imogen:~$ klist -f
klist: No credentials cache found (filename: /tmp/krb5cc_1000)
1 ansel@imogen:~$ sudo mount -v -t nfs4 -o sec=krb5p enya.colonelpanic.local:/mnt/foo /mnt/foo
[sudo] password for ansel: 
mount.nfs4: timeout set for Sun Jul  9 17:27:34 2023
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.2,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,minorversion=1,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting enya.colonelpanic.local:/mnt/foo
ansel@imogen:~$ 

==> /var/log/kerberos/krb5kdc.log <==
Jul 09 17:25:34 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: NEEDED_PREAUTH: nfs/[email protected] for krbtgt/[email protected], Additional pre-authentication required
Jul 09 17:25:34 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: ISSUE: authtime 1688919934, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, nfs/[email protected] for krbtgt/[email protected]
Jul 09 17:25:35 enya krb5kdc[71839](info): TGS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: ISSUE: authtime 1688919934, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, nfs/[email protected] for nfs/[email protected]

ansel@imogen:~$ kinit
Password for [email protected]: 
ansel@imogen:~$ sudo mount -v -t nfs4 -o sec=krb5p enya.colonelpanic.local:/mnt/foo /mnt/foo
mount.nfs4: timeout set for Sun Jul  9 17:29:21 2023
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.2,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,minorversion=1,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting enya.colonelpanic.local:/mnt/foo
32 ansel@imogen:~$ 

==> /var/log/kerberos/krb5kdc.log <==
Jul 09 17:26:53 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: NEEDED_PREAUTH: [email protected] for krbtgt/[email protected], Additional pre-authentication required
Jul 09 17:27:16 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: ISSUE: authtime 1688920036, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, [email protected] for krbtgt/[email protected]

ansel@imogen:~$ sudo -s
root@imogen:/home/ansel# klist -f
klist: No credentials cache found (filename: /tmp/krb5cc_0)
root@imogen:/home/ansel# mount -vvvv -t nfs4 -o sec=krb5p enya.colonelpanic.local:/mnt/foo /mnt/foo
mount.nfs4: timeout set for Sun Jul  9 17:32:32 2023
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.2,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,minorversion=1,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting enya.colonelpanic.local:/mnt/foo
root@imogen:/home/ansel# 

==> /var/log/kerberos/krb5kdc.log <==

==> /var/log/kerberos/kadmin.log <==

==> /var/log/kerberos/krb5lib.log <==

root@imogen:/home/ansel# kinit ansel
Password for [email protected]: 
root@imogen:/home/ansel# mount -vvvv -t nfs4 -o sec=krb5p enya.colonelpanic.local:/mnt/foo /mnt/foo
mount.nfs4: timeout set for Sun Jul  9 17:34:48 2023
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.2,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,minorversion=1,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5p,vers=4,addr=192.168.2.3,clientaddr=192.168.2.124'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting enya.colonelpanic.local:/mnt/foo
root@imogen:/home/ansel# 

==> /var/log/kerberos/krb5kdc.log <==

==> /var/log/kerberos/kadmin.log <==

==> /var/log/kerberos/krb5lib.log <==

==> /var/log/kerberos/krb5kdc.log <==
Jul 09 17:32:01 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: NEEDED_PREAUTH: [email protected] for krbtgt/[email protected], Additional pre-authentication required
Jul 09 17:32:11 enya krb5kdc[71839](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.2.124: ISSUE: authtime 1688920331, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, [email protected] for krbtgt/[email protected]

EDITED to add extra information as per @user1686's comment:

rpc.gssd is running on the client, and NEED_GSSD=yes amd RPCGSSDOPTS="-vvv" are specified in /etc/default/nfs-common on the client:

ansel@imogen:~$ ps aux | grep rpc.gssd
root         604  0.0  0.0  12980  2020 ?        Ssl  Jul10   0:00 /usr/sbin/rpc.gssd
ansel    1792798  0.0  0.0   9212  2344 pts/1    S+   15:15   0:00 grep --color=auto rpc.gssd
ansel@imogen:~$ grep -i gss /etc/default/nfs-common
# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=yes
RPCGSSDOPTS="-vvv"
ansel@imogen:~$ 

Where are the rpc.gssd logs?

The mount shows up in exportfs -v on the server, however I've noticed that both gss/krb5p and sec=sys are specified (I only specified gss/krb5p):

ansel@enya:~$ sudo exportfs -v
/mnt/foo        gss/krb5p(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
ansel@enya:~$ 

The server has rpc.mountd running:

ansel@enya:~$ ps aux | grep rpc.mountd
root      575641  0.0  0.0   5440  2712 ?        Ss   Jul09   0:00 /usr/sbin/rpc.mountd
ansel    1152704  0.0  0.0   6612  2444 pts/2    S+   15:47   0:00 grep --color=auto rpc.mountd
ansel@enya:~$ 

The only gss service running on the server is rpc.gssd, even though NEED_SVCGSSD is set to "yes" in /etc/default/nfs-kernel-server:

ansel@enya:~$ ps aux | grep gss
root       72192  0.0  0.0  12980  1552 ?        Ssl  Jul01   0:01 /usr/sbin/rpc.gssd
ansel    1157863  0.0  0.0   6608  2244 pts/2    R+   15:53   0:00 grep --color=auto gss
ansel@enya:~$ grep -i gss /etc/default/nfs-kernel-server
# Do you want to start the svcgssd daemon? It is only required for Kerberos
NEED_SVCGSSD="yes"
# Options for rpc.svcgssd.
RPCSVCGSSDOPTS="-vvv"
ansel@enya:~$ 

EDIT 2: It works!

I'm not sure why svcgssd isn't starting automatically, but after starting it manually with sudo service rpc-svcgssd start I appear to be able to mount:

ansel@enya:~$ sudo service rpc-svcgssd start
ansel@enya:~$ ps aux | grep gss
root       72192  0.0  0.0  12980  1552 ?        Ssl  Jul01   0:01 /usr/sbin/rpc.gssd
root     1190509  0.0  0.0   4876  3696 ?        Ss   16:31   0:00 /usr/sbin/rpc.svcgssd
ansel    1190606  0.0  0.0   6608  2296 pts/3    S+   16:31   0:00 grep --color=auto gss
ansel@enya:~$ 

ansel@imogen:~$ sudo mount -v -t nfs4 -o sec=krb5p enya.colonelpanic.local:/mnt/foo /mnt/foo                           
[sudo] password for ansel:                                                                                             
mount.nfs4: timeout set for Tue Jul 11 16:34:43 2023                                                                   
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.2,addr=192.168.2.3,clientaddr=192.168.2.124'                   
ansel@imogen:~$ mount | grep enya
enya.colonelpanic.local:/mnt/foo on /mnt/foo type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.2.124,local_lock=none,addr=192.168.2.3)
ansel@imogen:~$ 

Thanks for your help @user1686.

Ansel

The only gss service running on the server is rpc.gssd, even though NEED_SVCGSSD is set to "yes" in /etc/default/nfs-kernel-server:

The kernel requires either gssproxy or rpc.svcgssd to actually handle the server-side Kerberos "handshake" using the user-space libkrb5/libgssapi. (rpc.gssd is the client-side equivalent.)

(On recent systems gssproxy is preferred – but at least Debian's version of nfs-utils still seems to depend on the older rpc.svcgssd daemon.)

From your description, it sounds like the daemon does start up once (for the initial mount to work) but either crashes or is stopped for some reason. Check journalctl logs to find out why.

Previous:

At one point when receiving mount errors, I realised there were missing principals for enya.colonelpanic.local (although I set the Kerberos server for the realm to be krb.colonelpanic.local),

You've set the Kerberos server to be "krb...", not the NFS server – i.e. that's not what the missing principal errors were about.

Your REALM { kdc = ... } configuration in krb5.conf only specifies the KDC, i.e. the system that issues tickets – it has nothing to do with NFS. (In fact, the KDC is not considered a "host-based service" at all, so you don't even need to create host/ principals for it – and certainly not nfs/ principals!)

One thing I noticed is that I don't have a principal for kadmin/[email protected] or kadmin/[email protected].

That's normal; MIT Kerberos prefers the generic kadmin/admin instead, and has stopped creating the FQDN-based principals for kadmin as of Kerberos 1.19. (Docs say kadmin/<fqdn> was only ever added for interop with Solaris SEAM.)

/mnt/foo gss/krb5p(rw,sync,no_subtree_check)

Despite the default comment, this is practically obsolete syntax. Your /etc/exports should look like this – the security modes are specified via sec=, similar to client side, while the "client host" field works as usual:

/mnt/foo *(rw,sync,no_subtree_check,sec=krb5p)

The Debian Wiki article states that 'Additional pre-authentication required' errors are due to incorrect keytabs, but the keytabs are correct as far as I know:

This is a normal "error" in most cases – it's how pre-auth works. You can ignore it if it's immediately followed by a successful AS-REQ. (That is, it's the next request that indicates whether the keytab was incorrect.) Since your messages are followed by an ISSUE:, it means the keytab is okay.

(You can use kinit -k to determine whether a keytab has the correct key, much like you would check a password.)

Historically, you would ask the KDC (AS) for the 'krbtgt' ticket and the AS would just…give it to you – the security relied on the ticket's session key being encrypted with the password¹ that only you know, so if anyone else asked for the krbtgt in your name, they couldn't do anything with it, as without the password they could not unlock the session key they received.

But they could enact offline brute-force attacks on the received ticket (i.e. guessing the password as much as they wanted to, without such attempts showing up in the KDC logs at all), so "pre-authentication" was added to the existing AS-REQ process as a pseudo-error that would be understood by "new" clients as an invitation to retry the AS-REQ with some auth data attached.

So it's a lot like how web servers give you a 401 "error" code when a URL requires HTTP authentication (e.g. HTTP Basic) – the client recognizes this error as an invitation to send the auth details.

Principals that exclusively use keytabs don't strictly need to have the 'preauth' flag enabled (as it's infeasible to brute-force the randomly generated AES key), but it doesn't hurt either.

¹ (A keytab can be thought of as a "password".)