Ubuntu Server 20.04 cloud-config adding user does nothing

I am trying to create a VirtualBox .box image of Ubuntu Server 20.04 with packer. The box is to be later used by Vagrant. I want to have a single user (vagrant) with root privileges who logs through SSH with public/private key pair.

The configuration files are as follows:

ubuntu2004.pkr.hcl:

source "virtualbox-iso" "autogenerated_1" {
  boot_command            = ["<enter><wait2><enter><wait><f6><esc><wait>", "autoinstall<wait2> ds=nocloud;", "<wait><enter>"]
  boot_wait               = "2s"
  cd_files                = ["./http/user-data", "./http/meta-data"]
  cd_label                = "cidata"
  disk_size               = 8192
  guest_additions_path    = "VBoxGuestAdditions_{{ .Version }}.iso"
  guest_os_type           = "Ubuntu_64"
  headless                = false
  http_directory          = "http"
  iso_checksum            = "sha256:f8e3086f3cea0fb3fefb29937ab5ed9d19e767079633960ccb50e76153effc98"
  iso_urls                = ["https://releases.ubuntu.com/focal/ubuntu-20.04.3-live-server-amd64.iso"]
  shutdown_command        = "echo 'ubuntu'|sudo -S shutdown -P now"
  ssh_handshake_attempts  = "200"
  ssh_password            = "ubuntu"
  ssh_port                = 22
  ssh_username            = "ubuntu"
  ssh_wait_timeout        = "10000s"
  vboxmanage              = [["modifyvm", "{{ .Name }}", "--memory", "1024"], ["modifyvm", "{{ .Name }}", "--cpus", "1"]]
  virtualbox_version_file = ".vbox_version"
  vm_name                 = "packer-ubuntu-20.04-amd64"
}

build {
  sources = ["source.virtualbox-iso.autogenerated_1"]

  provisioner "file" {
    destination = "/home/vagrant/authorized_keys"
    source      = "/home/user/.ssh/virtual_id_ed25519"
  }

  provisioner "file" {
    destination = "/home/vagrant/.ssh/authorized_keys"
    source      = "/home/user/.ssh/virtual_id_ed25519"
  }

  provisioner "shell" {
    scripts = ["scripts/init.sh", "scripts/cleanup.sh"]
  }

  post-processor "vagrant" {
    compression_level = "8"
    output            = "ubuntu-20.04-<no value>.box"
  }
}

The cloud-config yaml is as follows:

./http/user-data:

#cloud-config
autoinstall:
  version: 1
  locale: en_US
  keyboard:
    layout: en
    variant: us
  network:
    network:
      version: 2
      ethernets:
        enp0s3:
          dhcp4: true
  storage:
    layout:
      name: lvm
  identity:
    hostname: ubuntu-server
    username: ubuntu
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
  ssh:
    install-server: yes
  groups:
    - ubuntu: [root, sys]
    - cloud-users
  users:
    - default
    - name: vagrant
      ssh_authorized_keys:
        - ssh-ed25519 <<my-public-key>>
      sudo: ALL=(ALL) NOPASSWD:ALL
      groups: sudo, users, admin
      lock_passwd: true
      shell: /bin/bash
  user-data:
    disable_root: false
  packages:
    - openssh-server
    - build-essential
  late-commands:
    - echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/ubuntu

I am explicitly uploading the public key to the image with the two file provisioners. Is this necessary?

The configuration in user-data should follow the documentation.

When Vagrant starts the machine, it can't log in with SSH. If I manually log into the server directly (with the ubuntu user), I can see that there is no user vagrant, i.e. getent passwd | grep vagrant returns nothing.

So,

1. How should I set up the cloud-config so that after the box is created I can log in with the user vagrant and with SSH keys (no passwords)?
2. Do I need the user ubuntu? If not, how can I remove it?
3. What is the identity part in user-data doing? Can I remove the password from there, use SSH keys and not bother with creating explicitly a new user with SSH keys?

I hope that these things can be done inside the configuration files and not via shell scripts.