OpenLDAP and Kerberos servers on Ubuntu Server 22.04; Krb5 isn't creating /var/lib/krb5kdc/principal

I have an existing openldap server on Ubuntu Server 22.04 and am trying to set up a kerberos server with it, following this guide: https://ubuntu.com/server/docs/service-kerberos-with-openldap-backend

The accounts are created and tested, they work fine (I reference them by cn, but they have a uid; I just created them in apache directory studio first). I've done "dpkg-reconfigure krb5-config" and edited /etc/krb5.conf to add my server and realm. One thing not in that doc but that I've seen elsewhere is to create a section "[domain_realm]" and add my domain to it, which I've done but it hasn't helped. I created a logging section, which requires modifying the systemd services to allow writing to /var/log/kerberos, but that's fine. I then run the "kdb5_ldap_util" command as listed, and after entering the passwords it appears to complete successfully. I do the stash for the kdc-service and kadmin-service passwords, that completes successfully. I create the kadm.acl file, that is fine, then when I try to start the service, I get this in krb5kdc.log:

krb5kdc[712216](Error): Cannot find master key record in database - while fetching master keys list for realm SUBDOMAIN.DOMAIN.COM

If I then run "kadmin.local" I get the following:

Authenticating as principal root/[email protected] with password.
kadmin.local: Cannot find master key record in database while initializing kadmin.local interface

As it turns out, the file /var/lib/krb5kdc/principal never gets created. I'm not sure what else to troubleshoot here, but at this point I'm stuck and any guidance would be appreciated. Config files below, with domain and subdomain changed:

/etc/krb5.conf:

[libdefaults]
        default_realm = SUBDOMAIN.DOMAIN.COM

# The following krb5.conf variables are only for MIT Kerberos.
        kdc_timesync = 1
        ccache_type = 4
        forwardable = true
        proxiable = true

# The following encryption type specification will be used by MIT Kerberos
# if uncommented.  In general, the defaults in the MIT Kerberos code are
# correct and overriding these specifications only serves to disable new
# encryption types as they are added, creating interoperability problems.
#
# The only time when you might need to uncomment these lines and change
# the enctypes is if you have local software that will break on ticket
# caches containing ticket encryption types it doesn't know about (such as
# old versions of Sun Java).

#       default_tgs_enctypes = des3-hmac-sha1
#       default_tkt_enctypes = des3-hmac-sha1
#       permitted_enctypes = des3-hmac-sha1

# The following libdefaults parameters are only for Heimdal Kerberos.
        fcc-mit-ticketflags = true

[logging]
        kdc = FILE:/var/log/kerberos/krb5kdc.log
        admin_server = FILE:/var/log/kerberos/kadmin.log
        default = FILE:/var/log/kerberos/krb5lib.log

[realms]
        SUBDOMAIN.DOMAIN.COM = {
                kdc = hostname.subdomain.domain.com
                admin_server = hostname.subdomain.domain.com
                default_domain = subdomain.domain.com
                database_module = openldap_ldapconf
        }

[domain_realm]
        .subdomain.domain.com = SUBDOMAIN.DOMAIN.COM
        subdomain.domain.com = SUBDOMAIN.DOMAIN.COM

[dbdefaults]
        ldap_kerberos_container_dn = cn=krbContainer,dc=subdomain,dc=domain,dc=com

[dbmodules]
        openldap_ldapconf = {
                db_library = kldap

                # if either of these is false, then the ldap_kdc_dn needs to
                # have write access
                disable_last_success = true
                disable_lockout  = true

                # this object needs to have read rights on
                # the realm container, principal container and realm sub-trees
                ldap_kdc_dn = "cn=kdc-service,ou=accounts,dc=subdomain,dc=domain,dc=com"

                # this object needs to have read and write rights on
                # the realm container, principal container and realm sub-trees
                ldap_kadmind_dn = "cn=kadmin-service,ou=accounts,dc=subdomain,dc=domain,dc=com"

                ldap_service_password_file = /etc/krb5kdc/service.keyfile
                ldap_servers = ldapi:///
                ldap_conns_per_server = 5
        }

/etc/krb5kdc/kadm5.acl:

*/[email protected] *

/etc/krb5kdc/kdc.conf:

[kdcdefaults]
    kdc_ports = 750,88

[realms]
    SUBDOMAIN.DOMAIN.COM = {
        database_name = /var/lib/krb5kdc/principal
        admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
        acl_file = /etc/krb5kdc/kadm5.acl
        key_stash_file = /etc/krb5kdc/stash
        kdc_ports = 750,88
        max_life = 10h 0m 0s
        max_renewable_life = 7d 0h 0m 0s
        master_key_type = des3-hmac-sha1
        #supported_enctypes = aes256-cts:normal aes128-cts:normal
        default_principal_flags = +preauth
    }

Edit: Per user1686's comment, I can see now that I have a potential access issue for the accounts. I'm a bit over my head on this piece, but using 'slapcat -b cn=config' I have the following olcAccess rules:

olcAccess: {0}to * by dn="cn=admin,dc=subdomain,dc=domain,dc=com" manage by dn="cn=surfrock66,ou=accounts,dc=subdomain,dc=domain,dc=com" manage by dn="cn=ldapbinduser,ou=accounts,dc=subdomain,dc=domain,dc=com" read by * break
olcAccess: {1}to dn.children="ou=accounts,dc=subdomain,dc=domain,dc=com" attrs=userPassword,shadowExpire,shadowInactive,shadowLastChange,shadowMax,shadowMin,shadowWarning by self write by anonymous auth
olcAccess: {2}to attrs=krbPrincipalKey by anonymous auth by dn.exact="cn=kdc-service,ou=accounts,dc=subdomain,dc=domain,dc=com" read by dn.exact="cn=kadmin-service,ou=accounts,dc=subdomain,dc=domain,dc=com" write by self write by * none
olcAccess: {3}to dn.subtree="cn=krbContainer,dc=subdomain,dc=domain,dc=com" by dn.exact="cn=kdc-service,ou=accounts,dc=subdomain,dc=domain,dc=com" read by dn.exact="cn=kadmin-service,ou=accounts,dc=subdomain,dc=domain,dc=com" write by * none
olcAccess: {4}to dn.subtree="dc=subdomain,dc=domain,dc=com" by self read

2022.12.08-08-36 PST Edit: So I am much further, and I did the following:

addprinc ldap/[email protected]
ktadd -k /etc/ldap/kerberos.ldap.hostname.subdomain.domain.com.keytab ldap/hostname.subdomain.domain.com
chown openldap:openldap /etc/ldap/kerberos.ldap.hostname.subdomain.domain.com.keytab
chmod 640 /etc/ldap/kerberos.ldap.hostname.subdomain.domain.com.keytab

I edited /etc/default/slapd and added this to it:

export KRB5_KTNAME="FILE:/etc/ldap/kerberos.ldap.hostname.subdomain.domain.com.keytab"

When I restarted the service, testsaslauthd still works, but ldapsearch -D does not. I'm wondering if there's a config change I need to make in ldap? From one of the resources I was working off of, I constructed this ldif for config, but ultimately I don't think I understand what it will do and if it was right:

dn: cn=config
changetype: modify
add: olcSaslHost
olcSaslHost: hostname.subdomain.domain.com
-
add: olcSaslRealm
olcSaslRealm: SUBDOMAIN.DOMAIN.COM
-
add: olcAuthzRegexp
olcAuthzRegexp: {0}"cn=([^/]*),cn=subdomain.domain.com,cn=GSSAPI,cn=auth" "cn=$1,ou=accounts,dc=subdomain,dc=domain,dc=com"
-
add: olcAuthzRegexp
olcAuthzRegexp: {1}"cn=host/([^/]*).subdomain.domain.com,cn=subdomain.domain.com,cn=GSSAPI,cn=auth" "cn=$1,ou=hosts,dc=subdomain,dc=domain,dc=com"
-
add: olcAuthzRegexp
olcAuthzRegexp: {2}"uid=ldap/admin,cn=subdomain.domain.com,cn=GSSAPI,cn=auth" "cn=admin,dc=subdomain,dc=domain,dc=com"

I think this is the last missing step? Something in the config pointing LDAP to sasl? Or am I approaching this all wrong?

principal is the file that contains the DB2-format KDC database. It is not created on your system because you're not using the db2 backend – you're using LDAP, which replaces all local storage for the KDC – so a local database file is unnecessary and irrelevant to the problem you're having.

If you've run kdb5_ldap_util create according to instructions, you should check via LDAP whether it has actually created the necessary entries – there should be, at minimum, an entry for K/M@<realm> and another for krbtgt/<realm>@<realm>. The error message is about the "K/M" entry, which is a pseudo-principal holding the database mkey (which is encrypted with the "master key" password).

Make sure to check LDAP using the KDC's account, just in case it hasn't been granted the necessary privileges to read the Kerberos LDAP objects. (The accounts don't need to have an uid; they're only used as LDAP bind accounts, not as local Unix accounts.)

As Ubuntu sometimes uses AppArmor, be sure to check dmesg for AVC denial messages – it could be that the KDC isn't allowed to read "service.keyfile" or to access your OpenLDAP socket.

Additionally, enable the stats loglevel in your LDAP server to have it log all queries – this could e.g. reveal that the KDC is looking under a different DN (normally "ldap_kerberos_container_dn" would be set next to other dbmodules parameters).

Finally, if strace kadmin.local shows that the tool isn't actually connecting to LDAP but trying to access a local db2 file instead, then it's possible that you have configuration in kdc.conf (the KDC-specific config file) that overrides the system-wide settings in krb5.conf. Normally the KDC configuration is loaded from /var/lib/krb5kdc/kdc.conf and has priority over the krb5.conf global settings.

(Additionally, if you were using db2, the principal database still would not be created automatically on first startup – this needs to be done manually by running kdb5_util create [-s] and providing the master encryption key, just like you've done for the LDAP backend using "kdb5_ldap_util create".)

One thing not in that doc but that I've seen elsewhere is to create a section "[domain_realm]" and add my domain to it, which I've done but it hasn't helped.

This is only needed for Kerberos clients when obtaining tickets for a hostname which doesn't map 1:1 to its realm (the typical example being mit.edu → ATHENA.MIT.EDU). It has no relevance to the KDC's startup.

I created a logging section, which requires modifying the systemd services to allow writing to /var/log/kerberos,

It doesn't if you log to SYSLOG instead, which was likely the intent of whoever wrote the systemd services for Ubuntu.

isn't that not there since my storage is in LDAP?

No, the /etc/krb5.keytab file is not part of the KDC database – it belongs to the host as a "domain member" and stores the equivalent of machine account's Kerberos password. Each machine in your Kerberos realm should have its own /etc/krb5.keytab with at least the host/<fqdn> principal.

I'm trying to get LDAP passthrough auth working so the kerberos password is the real one; I'm finding a lot of the documentation is older. I see most people recomment saslauthd which I think I set up, but now it's looking for /etc/krb5.keytab

For OpenLDAP, yes, generally you need saslauthd.

1. When binding to a DN which has userPassword: {SASL}foo@BAR, slapd will use the libsasl "password check" functions.
2. Depending on the configured pwcheck_method, libsasl will contact saslauthd to verify the password.
3. saslauthd will use the backend configured through its -a option to do the actual verification.
4. With the -a krb5 backend, saslauthd will try to get an initial Kerberos ticket from a KDC with the provided username and password (equivalent to kinit).
5. If an initial ticket was successfully acquired, saslauthd will additionally request a service ticket for itself and will attempt to decrypt it using the machine's keytab.
6. If the service ticket could be acquired and decrypted, the password is accepted.

The last step is needed to prevent KDC spoofing attacks. Traditionally Kerberos KDCs issued tickets without validating your password (wrong password just meant the client couldn't decrypt the received ticket) and a malicious KDC can still do that – "preauth" is not mandatory at protocol level. So saslauthd needs to do the same thing any other kerberized service does in order to verify that the ticket is legitimate – attempt to decrypt it using a service keytab. (Both pam_krb5 and even Windows do the same during AD logons.)

I made host/hostname.fqdn@REALM and ldap/hostname.fqdn@REALM. I created a keytab for ldap; I converted 1 user's pass to "{SASL}name@REALM". Using testsaslauthd on that user, I get success. Using ldapsearch on that user, invalid creds. I think ldap isn't talking to saslauthd? Under "Kerberos Authentication" here I added the configs: help.ubuntu.com/community/OpenLDAPServer but when I do "ldapsearch -Y GSSAPI" I get "GSSAPI Error: No credentials were sup...

-Y GSSAPI is direct Kerberos authentication – completely separate from saslauthd and password passthrough. That is, slapd doesn't even receive a Kerberos password – it only receives a Kerberos ticket and validates it internally against its service keytab.

1. For SASL GSSAPI, the slapd service itself – not saslauthd – needs a keytab for ldap/<fqdn>. (By default all services look for their keys in the system /etc/krb5.keytab, but it might be better for security to keep things separate and set slapd's KRB5_KTNAME to a different path – that way you don't need to give it read access to the system keytab.)

2. The client also needs to know that it's meant to request a ticket for ldap/<fqdn>, so connecting to an IP address likely won't work (unless it has proper rDNS set) – you should be connecting to ldap://<fqdn> specifically. (It's a very similar issue to TLS SNI or hostnames in TLS certificates).

   If in doubt, export KRB5_TRACE=/dev/stderr before calling the client to figure out what principal it is trying to make a TGS-REQ for (or look at KDC logs, or a network capture).

3. The client must already have the initial Kerberos ticket (krbtgt/REALM) present, that is, you must kinit beforehand. If you don't have a TGT already, Kerberos clients will not ask for your password – they'll just immediately fail to authenticate.

Password passthrough is only relevant to "simple" binds (ldapsearch -D <user_dn> -W) and to SASL PLAIN binds. Test "simple" bind first (you can mostly ignore SASL PLAIN).

You must install libsasl2 on your OS and give ownership over your krb5.keytab to the openldap user.

on debian:

sudo apt install libsasl2-2 libsasl2-modules-gssapi-mit

and then

sudo chown openldap:openldap /etc/krb5.keytab or other keytab

Edit: but be careful with multiple keytabs on the same machine/container (with multiple servers).