SSH public key authentication with google authenticator still asks for password

I'm trying to enable 2FA with ssh using libpam-google-authenticator. Not all users need authenticator enabled. Everybody uses ssh public keys, and nobody has a password. I'm running Debian buster, and I've also tried libpam-google-authenticator from bullseye.

My problem is that no matter what I put in the PAM config, users without authenticator enabled are never logged straight in, but always asked for a password.

I've install libpam-google-authenticator and configured /etc/ssh/sshd_config with:

PasswordAuthentication no
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
PasswordAuthentication no
PermitEmptyPasswords no

I haven't been able to work out the correct PAM config so that users without a .google_authenticator file are still logged in. Depending on what I use, users are either prompted for a password (they don't have one), or not allowed in at all.

In /etc/pam.d/sshd I've tried (like this Trying to get SSH with public key (no password) + google authenticator working on Ubuntu 14.04.1):

#@include common-auth
auth       required     pam_google_authenticator.so debug nullok

In this case, users without an authenticator setup get rejected with the following debug;

Aug 05 15:11:18 <host> sshd(pam_google_authenticator)[746624]: debug: start of google_authenticator for "<user>"
Aug 05 15:11:18 <host> sshd(pam_google_authenticator)[746624]: debug: end of google_authenticator for "<user>" Result: The return value should be ignored by PAM dispatch
Aug 05 15:11:18 <host> sshd[746620]: error: PAM: Permission denied for <user> from <IP>

Is pam_permit is needed to set up the fallback case?

I've also tried various combinations of auth required and auth sufficient before and after @include common-auth but they all result in users without authenticator being asked for a password and sometimes users WITH authenticator also being asked for a password.

Does anyone have a recipe to make this work?

I don't think you need or want to comment out the @include common-auth. Or at least I did not and it seemed to work correctly. But I am still mostly been testing this.

Does anyone have a recipe to make this work?

Don't have time to translate it to a shell script for you, but this is a excerpt of an ansible playbook that seems to work for me. I suspect you should be able follow along with what this is doing even if you aren't using ansible.

- hosts: linux_systems
  tasks:

  - name: Add group 'totp'
    group:
      name: totp
      state: present
      system: yes

  - name: Create directory for totp secrets
    file:
      state: directory
      path: /var/lib/google-authenticator
      owner: "0"
      group: "0"
      mode: "0700"

  - name: install libpam-google-authenticator
    apt:
      update_cache: yes
      cache_valid_time: '{{ apt_cache_valid_time | default(7200) }}'
      state: present
      name:
      - libpam-google-authenticator

  - name: Create secret for 'example-user'
    args:
      creates: /var/lib/google-authenticator/example-user
    shell: |
      TOTP_USER=example-user; \
      google-authenticator \
        --force --quiet \
        --emergency-codes=10 \
        --time-based \
        --qr-mode=none \
        --allow-reuse \
        --window-size=3 \
        --rate-limit=4 --rate-time=30 \
        --secret=/var/lib/google-authenticator/${TOTP_USER}

  - name: update pam
    lineinfile:
      insertafter: '^@include common-password'
      path: /etc/pam.d/login
      line: 'auth required pam_google_authenticator.so nullok user=root secret=/var/lib/google-authenticator/${USER}'

  - name: update pam
    lineinfile:
      insertafter: '^@include common-password'
      path: /etc/pam.d/sshd
      line: 'auth required pam_google_authenticator.so nullok user=root secret=/var/lib/google-authenticator/${USER}'

  - name: update sshd ChallengeResponseAuthentication
    notify: Restart sshd
    lineinfile:
      path: /etc/ssh/sshd_config
      regexp: '^ChallengeResponseAuthentication .*'
      line: 'ChallengeResponseAuthentication yes'

  handlers:

  - name: Restart sshd
    service:
      name: sshd
      state: restarted