Join Log in
Score:1
Rino Bino avatar Ubuntu

Error while using private Apt repos during autoinstall installation

ae flag
Rino Bino

Overview / Scenario

I already have a fully working autoinstall configuration that is using the default apt configuration (no apt config in cloud-init user data). The source image is using the Ubuntu 20.04 cloud image.

My goal is to edit the autoinstall config to use private Apt repos (the repos already exist). I must ensure the installer never reaches out to Ubuntu servers for Apt packages and only hits custom Apt repos. The repos are already established and have been working for a long time, I just need to get the installer to point to them.

I have already reviewed How to add apt repository with Ubuntu autoinstall

Here is what I've added to the autoinstall cloud init:

Note: the below mentioned /cdrom/apt-key.gpg file is present and valid for the repo (tested thoroughly).

Note: The repos are over https using private CA. Since I can't install the ca-certificates package yet during install before configuring Apt, I'm just forcing it to be trusted. trusted=yes didn't work, I had to also add the apt.conf.d/99verify-peer.conf as shown below to get it to work. This may be a cause for the issue but it works fine manually at an install shell. I have the certs and can inject them into the install media, so if there is a better way to get the installer instance to trust my CA cert without having to force trust, I'm open to that change!

  early-commands:
    - 'mkdir -p /etc/apt'
    - 'cp /cdrom/apt-key.gpg /etc/apt/fresh-ubuntu-key.gpg'
    - 'touch /etc/apt/apt.conf.d/99verify-peer.conf'
    - 'echo >>/etc/apt/apt.conf.d/99verify-peer.conf "Acquire { https::Verify-Peer false }"'
  apt:
    preserve_sources_list: false
    sources_list: |
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-ver221 main
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-security-ver221 main
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-updates-ver221 main
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-backports-ver221 main

Problem / Troubleshooting

When trying to install, it fails with the following:

finish: subiquity/Updates/apply_autoinstall_config
start: subiquity/Late/apply_autoinstall_config
finish: subiquity/Late/apply_autoinstall_config
start: subiquity/Shutdown/apply_autoinstall_config
finish: subiquity/Shutdown/apply_autoinstall_config
finish: subiquity/apply_autoinstall_config
start:   subiquity/Install/install/configure_apt: configuring apt
start: subiquity/Meta/status_GET
start: subiquity/Meta/status_GET
start: subiquity/Meta/status_GET
start: subiquity/Meta/status_GET
start: subiquity/Mirror/cmd-apt-config: curtin command apt-config
finish: subiquity/Mirror/cmd-apt-config: curtin command apt-config
start:     subiquity/Install/install/configure_apt/cmd-in-target: curtin command in-target
finish:   subiquity/Install/install/configure_apt: Command '['/snap/subiquity/3119/usr/bin/python3.8', '-m', 'curtin', '--showtrace', '-vvv', '--set', 'json:reporting={"subiquity": {"type": "journald", "identifier": "curtin_event.2752.2"}}', 'in-target', '-t', '/tmp/tmpzyqgk825/mount', '--', 'apt-get', 'update']' returned non-zero exit status 100.
finish: subiquity/Install/install: Command '['/snap/subiquity/3119/usr/bin/python3.8', '-m', 'curtin', '--showtrace', '-vvv', '--set', 'json:reporting={"subiquity": {"type": "journald", "identifier": "curtin_event.2752.2"}}', 'in-target', '-t', '/tmp/tmpzyqgk825/mount', '--', 'apt-get', 'update']' returned non-zero exit status 100.
start:   subiquity/ErrorReporter/1669932701.344982386.install_fail/add_info
finish:     subiquity/Install/install/configure_apt/cmd-in-target: curtin command in-target
finish:   subiquity/ErrorReporter/1669932701.344982386.install_fail/add_info: written to /var/crash/1669932701.344982386.install_fail.crash
An error occurred. Press enter to start a shell

I tried finding anything meaningful in /var/crash/1669932701.344982386.install_fail.crash but it's a massive file and hard to parse. Doesn't seem to have any additional failure info that I can see.

Important Note: After it fails, I can drop into a shell, update /etc/apt/sources.list manually and apt update works perfectly fine. So I know that the installer instance CAN properly communicate with my private repos. So it seems the issue has to be something specific during autoinstall/cloud-init.

Please help if you have any ideas, thanks!!

1092
1 + 0
apt
Score:1
Andrew Lowther avatar Ubuntu
jp flag
Andrew Lowther

tl;dr

You could try something like

#cloud-config
autoinstall:
  apt:
    preserve_sources_list: false
    sources_list: |
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-ver221 main
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-security-ver221 main
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-updates-ver221 main
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-backports-ver221 main
     conf: |
      Acquire::https::repo.internal.example.com::Verify-Peer "false";
      Acquire::https::repo.internal.example.com::Verify-Host "false";

reasoning

To start, the early-commands in your autoinstall config are not doing what you seem to think. The commands are configuring apt within the installer environment. However, the apt commands during installation generally happen in a different chroot environment. The logs you shared show this. The failing command is partially in-target -t /tmp/tmpzyqgk825/mount apt-get update.

The second issue is that the repository has a custom gpg key. The only way to provide a custom key that I am aware of is using the syntax in the other answer you referenced. You can certainly also add the sources.list option trusted=yes to avoid a failure from the missing gpg key, but that is probably not a good configuration in the long run.

The third issue is that a private CA signed the https certificate. There is no provided way to trust this private CA. It looks like a bug has been filed requesting this. It also looks like there is an apt configuration option that can be used to ignore the certificate problems. Again, that is probably not a good configuration in the long run.

It is probably easiest to late-commands to install the custom gpg key and the private CA, and reconfigure apt.

update

As noted in the comments, this solution did not work. I'm sharing what I found here. I tested the install process using Ubuntu 22.04 (subiquity 22.04.2).

  • subiquity creates /var/log/installer/subiquity-curtin-apt.conf. It basically contains the autoinstall apt section (and potentially proxy information).
  • subiquity runs curtin to configure apt using /var/log/installer/subiquity-curtin-apt.conf in the /tmp/tmpXXX overlay file system. If my autoinstall snippet above is used then curtin will create /tmp/tmpXXX/etc/apt/apt.conf.d/94curtin-config.
  • subiquity creates /var/log/installer/subiquity-curtin-install.conf. This does not contain an apt section. Presumably, because apt is configured already.
  • subiquity runs curtin to perform the install using /var/log/installer/subiquity-curtin-install.conf. This will copy /tmp/tmpXXX to /target. This creates /target/etc/apt/apt.conf.d/94curtin-config.
  • At a later stage during the install, curtin configures apt. Because the install configuration does not contain the apt section curtin will actually delete 94curtin-config.
  • Subsequent apt commands that require the apt configuration in 94curtin-config will fail and cause the installation to fail.
  • If the installation does not fail because of the missing 94curtin-config file then a later step in subiquity will copy /tmp/tmpXXX/etc/apt to /target/etc/apt and the resulting installation will contain 94curtin-config.

In short, do not depend on the apt conf settings using autoinstall.

+ 5
Rino Bino avatar
ae flag
Rino Bino
Thank you! The key takeaway here is that `early-commands` wasn't working as I was expecting because the installation is working within a chroot directory. That explains the behavior and I was able to get this working using your config example. I'll use the custom gpg key but ignore the https validation. Thanks again for the help.
0
Reply
Rino Bino avatar
ae flag
Rino Bino
To anyone reading this in the future, I still could not get this to work because the `Verify-Peer` and `Verify-Host` config overrides are seemingly ignored during the install process (maybe apt-transport-https isnt available?). Instead of fighting it more I just updated the repo to accept `http://` connections.
0
Reply
Andrew Lowther avatar
jp flag
Andrew Lowther
I tested again and the installer did fail, but it fails much later than without the `conf` setting. The apt configuration options do get installed into `/tmp/tmpXXX/mount/etc/apt/apt.conf.d/94curtin-config`, but that does not get copied over to `/target/etc/apt/apt.conf.d/`. Probably a gap in _subiquity_'s logic. In general, I do think it is much easier to use a basic apt configuration and then change settings in `late-commands`. The installer simply does not have enough hooks to allow customization like installing gpg keys, installing trusted certificates, or adding apt configuration.
1
Reply
Andrew Lowther avatar
jp flag
Andrew Lowther
I did some more digging into why this fails. Unfortunately, it did not lead to a solution. I dumped what I found into the answer in case anybody else tries something similar.
1
Reply
Rino Bino avatar
ae flag
Rino Bino
Your effort is appreciated!
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.