Troubleshooting Ubuntu autoinstall for desktop with LUKS

I'm trying to troubleshoot an Ubuntu autoinstaller using cloud-init & autoinstall. Ideally I'm trying to have it encrypt & partition the main OS disk on a laptop after the installer runs. Everything in the following file is working, except for the Storage section, which I've copied over after running through the installer with Storage as interactive.

Here is the user-data file which I put together after reviewing a few different resources:

autoinstall:
  version: 1
  refresh-installer:  # start with an up-to-date installer
  update: yes
  channel: stable     # update from the ubuntu stable branch only
  interactive-sections:  # Install groups listed here will wait for user input
    - network
    - identity

  locale: en_US.UTF-8
  keyboard:
    layout: us
  ssh:
    allow-pw: false
    install-server: false
  storage:
    config:
    - grub_device: false
      id: disk-nvme0n1
      name: ''
      path: /dev/nvme0n1
      preserve: false
      ptable: gpt
      serial: KINGSTON SNVS1000G_50026B7784415179
      type: disk
      wipe: superblock
      wwn: eui.00000000000000000026b77844151795
    - device: disk-nvme0n1
      flag: boot
      grub_device: true
      id: partition-0
      number: 1
      preserve: false
      size: 536870912
      type: partition
      wipe: superblock
    - fstype: fat32
      id: format-0
      preserve: false
      type: format
      volume: partition-0
    - device: disk-nvme0n1
      flag: ''
      grub_device: false
      id: partition-1
      number: 2
      preserve: false
      size: 1073741824
      type: partition
      wipe: superblock
    - fstype: ext4
      id: format-1
      preserve: false
      type: format
      volume: partition-1
    - device: disk-nvme0n1
      flag: ''
      grub_device: false
      id: partition-2
      number: 3
      preserve: false
      size: 998591430656
      type: partition
      wipe: superblock
    - id: dm_crypt-0
      keyfile: /tmp/luks-key-em_r7gfw
      preserve: false
      type: dm_crypt
      volume: partition-2
    - devices:
      - dm_crypt-0
      id: lvm_volgroup-0
      name: ubuntu-vg
      preserve: false
      type: lvm_volgroup
    - id: lvm_partition-0
      name: ubuntu-lv
      preserve: false
      size: 107374182400B
      type: lvm_partition
      volgroup: lvm_volgroup-0
      wipe: superblock
    - fstype: ext4
      id: format-2
      preserve: false
      type: format
      volume: lvm_partition-0
    - device: format-2
      id: mount-2
      path: /
      type: mount
    - device: format-1
      id: mount-1
      path: /boot
      type: mount
    - device: format-0
      id: mount-0
      path: /boot/efi
      type: mount
    version: 1

  apt:
    preserve_sources_list: false
    primary:
    - arches: [amd64, i386]
      uri: http://us.archive.ubuntu.com/ubuntu
    - arches: [default]
      uri: http://ports.ubuntu.com/ubuntu-ports
    conf: | # APT config
      APT {
        Get {
          Assume-Yes "true";
          Fix-Broken "true";
        };
      };
    sources:
      ignored1:  # for yaml formatting. below adds graphics drivers ppa
        source: ppa:graphics-drivers/ppa

  snaps:
    - name: slack
      classic: true
      channel: stable

  packages: 
    - build-essential
    - ubuntu-desktop
    - dkms
    # Zoom dependencies
    - libgl1-mesa-glx
    - libegl1-mesa
    - libxcb-xtest0
    - libxcb-xinerama0
    # - wormhole for an easy to use encrypted file transfer
    - wormhole
    - emacs
    #- Commented out desktop environments for future selection menu via grub
    #- ubuntu-mate-desktop
    #- xfce4
    #- kde-full
    - git
    - libreoffice
    # pavucontrol for bluetooth/pulseaudio
    - pavucontrol
    #- gnome-session - used for ubuntu-minimal-desktop
    #- gdm3 - ubuntu-minimal-desktop
    # ClamAV packages
    - clamav
    - clamtk
    #- clamtk-gnome
    - clamav-daemon
    - clamav-docs
    # - chrome & some dependencies for the user experience
    - google-chrome-stable
    - chromium-codecs-ffmpeg-extra
    # Bluetooth dependencies
    - bluez-btsco
    - bluez-dbg
    - bluez-hcidump
    - bluez-tools
    - libbluetooth3-dbg
    # End of Bluetooth dependencies
    # These packages are for adding a repo that requires https, in particular for Sublime Text which we install later
    - dirmngr
    - gnupg
    - apt-transport-https
    - ca-certificates
    - software-properties-common
    - sublime-text
    # End https repo/sublime dependencies
    # Uncomment for OEM kernel
    #- linux-oem-20.04b

  package_update: true
  package_upgrade: true

  late-commands:
    # Transfer RHR wallpaper from USB to user backgrounds
    # Changing from networkd to NetworkManager
    # move existing config out of the way
    - find /target/etc/netplan/ -name "*.yaml" -exec sh -c 'mv "$1" "$1-orig"' _ {} \;
    # Create a new netplan and enable it
    - |
      cat <<EOF | sudo tee /target/etc/netplan/01-netcfg.yaml
      network:
        version: 2
        renderer: NetworkManager
      EOF
    - curtin in-target --target /target netplan generate
    - curtin in-target --target /target netplan apply
    - curtin in-target --target /target systemctl enable NetworkManager.service
    #- cp /target/cdrom/extras/desktop-wp.png /target/usr/share/backgrounds/
  
  user-data: # Commands here run during first boot (cannot be interactive)
    runcmd:  # The script below will be dropped in /var/lib/cloud/instance/scripts/runcmd.sh and will be executed at first boot
      - [ apt-get, update ]
      - [ apt-get, dist-upgrade, --yes ]
      # Install Google Chrome from the official repository setup earlier
      - [ apt, install, --assume-yes, google-chrome-stable ]
      # Make a directory for additional post-installers
      - [ mkdir, -p, /run/post-install/ ]
      # Download & install Zoom
      - [ wget, "https://zoom.us/client/latest/zoom_amd64.deb", -O, /run/post-install/zoom_amd64.deb]
      - [ dpkg, -i, /run/post-install/zoom_amd64.deb ]
      - [ apt, --fix-broken, --assume-yes, install ]
      # Remove packages we don't need
      - apt-get --assume-yes purge firefox thunderbird byobu
      - apt autoremove --assume-yes
      #- [ sudo, -u, ubuntu, dbus-launch, gsettings, set, org.gnome.desktop.background, picture-uri, file:///usr/share/backgrounds/desktop-wp.png]
      - apt update
      - apt install --assume-yes sublime-text
      # Add chrome & sublime text to favorites (currently not working)
      - gsettings set org.gnome.shell favorite-apps "$(gsettings get org.gnome.shell favorite-apps | sed s/.$//), 'google-chrome.desktop', 'sublime_text.desktop']" 
      - gsettings set org.gnome.shell.extensions.dash-to-dock click-action 'minimize'

I've tried commenting out the "serial" section in Storage to see if that would make a change, but even copying the output from the curtin-install-cfg.yaml for the Storage section doesn't seem to work.

I found this bug report that seems to imply something simpler may come in the future but isn't here yet: https://bugs.launchpad.net/subiquity/+bug/1913986

Referenced the following in building this out:

• https://ubuntu.com/server/docs/install/autoinstall-reference#storage
• https://curtin.readthedocs.io/en/latest/topics/storage.html
• https://www.pugetsystems.com/labs/hpc/How-To-Make-Ubuntu-Autoinstall-ISO-with-Cloud-init-2213/

The PGP keys for Chrome & Sublime-Text have been omitted due to length but they're working. Just can't get the storage setup!

Any guidance is appreciated!

The problem with your user-data file is likely that you are specifying the "keyfile" where the encryption password is stored after installation. That /tmp directory is where the key gets stored after installation and likely changes every time you autoinstall.

Instead of specifying the keyfile dir in your yaml try specifying the actual passphrase with the "key" key.

key:

The key key specifies the password of the encryption key. The target system will prompt for this password in order to mount the disk.

keyfile:

The keyfile contains the /tmp dir of the password for the encryption key.

Exactly one of key and keyfile must be supplied.

Example:

- id: lvm_partition_1
  type: dm_crypt
  dm_name: crypto
  volume: sdb1
  key: changeme