Webmin backup via ssh not working with remote rush shell

I want to make backups from Webmin, initially the configuration backups, to a remote server securely. I'm new to Webmin and to rush. This seems to be a config issue with Webmin, rush, or sshd.

My Backup storage server 10.20.0.130 (Ubuntu 20.04) has an account ftpsecure with shell /usr/sbin/rush.

/etc/rush.rc is as follows:

  acct on
  limits t10r20
  umask 002
  env - USER LOGNAME HOME PATH
  fall-through
rule pwd
  command ^pwd
  set[0] /bin/pwd
rule scp-to
  command ^scp (-v )?(-r )?-t( --)? /srv/vmceml_backups/?
  set[0] /usr/bin/scp
  chroot /srv/rush
  chdir /srv/vmceml_backups

From my webmin server 10.20.0.129 (Webmin 2.013 on Ubuntu 18.04), I can successfully run:

sudo -u ftpsecure scp -v testfile [email protected]:/srv/vmceml_backups/testfile

No password is requested which proves that the public/private key setup is correct and working, also the verbose log from scp confirms that.

Under Webmin, Backup Configuration Files, Scheduled Backups I have an entry: Backup destination: SSH server 10.20.0.130; file on server: /srv/vmceml_backups/webmin-vmceml-backups-%Y%m%d.tgz; Login as user ftpsecure; Server port default; Include: Webmin module config files, Server config files.

When I click "Save and Backup Now" I get: Starting backup of 45 modules to /srv/vmceml_backups/webmin-vmceml-backups-20230307.tgz on 10.20.0.130 via SSH .. scp failed : [email protected]: Permission denied (publickey).

On the backup storage server in /etc/ssh/sshd_config I have:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
LogLevel DEBUG
PermitRootLogin prohibit-password
StrictModes yes
AuthorizedKeysFile .ssh/authorized_keys
HostbasedAuthentication no
IgnoreRhosts yes
PasswordAuthentication no
PermitEmptyPasswords no
PubkeyAuthentication yes
TCPKeepAlive yes
Match User ftpsecure
        PasswordAuthentication no
        ForceCommand internal-sftp
        PermitTunnel no
        AllowAgentForwarding no
        AllowTcpForwarding no
        X11Forwarding no
Match all
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp  /usr/lib/openssh/sftp-server

Note: the presence/absence of the ForceCommand under Match User ftpsecure, or indeed the presence/absence of the entire section Match User ftpsecure, makes no difference.

I have not altered the standard configuration files for PAM.

The restricted shell rush on the backup storage server doesn't log anything when I run the Webmin backup, not even for the default rule. When I run scp manually on the webmin server then rush on the backup storage server logs as expected against the scp-to rule (and the scp succeeds, as noted above).

I don't understand the SSHD logs:

KEX done [preauth]
userauth-request for user ftpsecure service ssh-connection method none [preauth]
attempt 0 failures 0 [preauth]
user ftpsecure matched 'User ftpsecure' at line 70
PAM: initializing for "ftpsecure"
PAM: setting PAM_RHOST to "10.20.0.129"
PAM: setting PAM_TTY to "ssh"
Connection closed by authenticating user ftpsecure 10.20.0.129 port 51584 [preauth]
do_cleanup [preauth]

I haven't been able to find out what scp command exactly, is used by Webmin for these backups, if I knew that I could be sure my manual test corresponds to it.

Any suggestions welcome please!

I’ve seen this issue also when I moved to key based authentication. I’ve noticed a virtual in user suggest that this can be solved by running the backup command manually and adding a colon after the username on the scp command however I can locate a perl file to manually edit to get automatic backups working yet.