Join Log in
Score:1
Stephen Winnall avatar Ubuntu

How to use `apt` to update different layers of an Ubuntu installation on a union filesystem

cn flag
Stephen Winnall

I have installed Ubuntu 22.04 on a union filesystem on my NAS and boot it on diskless clients using NFS. The lowest layer in the filesystem contains a minimal, bootable version of Ubuntu. I can boot that layer on its own and update it using apt.

I would like the upper layer in the union filesystem to contain additional packages not included in the lower layer. By booting the union filesystem consisting of lower and upper layers with the upper layer read-write, I can install those additional packages using apt.

My problems begin when I have to update packages on the lower filesystem. I use apt for this and it seems to work.

Having updated the lower filesystem, I return to the union filesystem and try to update the upper filesystem. However, apt appears to want to update the packages from the lower filesystem again too, in addition to the ones in the upper filesystem. This would result in duplication on the upper and lower filesystems, which I don't want.

I have tried cleaning out /var/lib/apt and /var/cache/apt before doing apt update on the upper filesystem, but this doesn't solve the problem. This is partly because the packages on the upper filesystem are referencing versions on the lower filesystem that have been updated, and therefore appear not to exist anymore. So apt kindly tries to re-install them, but on the upper filesystem.

I thought about just doing an apt upgrade on the upper filesystem and then going through the duplicated packages and forcibly purging them somehow, but I haven't found a strategy yet.

I suspect that I am way outside the scope of what apt was designed for. Are there any apt gurus out there who can provide me with some guidance? The hacker in me is pondering attacking /var/lib/dpkg/* with a Perl script...

Additional Information

The directory /var/lib/dpkg/info/ occurs in both the lower and the upper layers. It appears that the version in the each layer only contains information for the packages installed in the same layer. Since the information for each for each package is held in separate files specific to that package, merging the two directories in the union filesystem results in a single view of all installed packages, which is what I had hoped for.

However, /var/lib/dpkg/status is a single text file that contains information about all packages, and the only visible merged version is from the upper layer. Any later changes to the lower layer are not copied up and thus are not visible. This causes problems in the upper layer which contains references to stuff that has been updated and has effectively disappeared.

Do I need to rebuild /var/lib/dpkg/status?

A Step Back, but also More Detail

[This is provided in anticipation of my anticipated attempt at answering my question myself.]

I'm going to try to answer this myself. Please feel free to comment and point out any errors or false assumptions that I make.

To recap: I am trying to create and maintain Ubuntu images that can be booted via NFS on (many different) diskless devices. The Ubuntu images are stored on and exported from a central NAS as union filesystems. My NAS only supports aufs, so that is what I use here.

Each image is constructed with aufs as follows on the NAS:

mount -t aufs o=br:<instance-data>=rw \
                  :<application>=ro \
                  :<local-Ubuntu-config>=ro \
                  :<Canonical-Ubuntu-image>=ro
              none
              <merged-image>            # e.g. /mnt/xxx

Each aufs image is exported by NFS (yes, that is possible).

There is one of these images for each diskless client. Each image has 4 layers, defined as branches (o=br:...). Since ogres have layers, I call these ogres.

Each layer has access to the layers below it, so it can use any packages and configuration installed there. Each layer also has its own additional packages and configuration.

All ogres share the same <Canonical-Ubuntu-image> and the same <local-Ubuntu-config>. These two layers could be collapsed into a single layer, but I want to keep a clear distinction between Canonical's stuff and my local configuration.

There are a small number of <application>s (e.g. network management, network monitoring, smart home aka OpenHAB, development, ...). They all share the same and`, as mentioned above, but have their own application layer.

Some ogres are duplicated (e.g. network management, OpenHAB) so they share the same application but need individual configuration for their own instance.

Excuse Me, but Why?

At the moment I have about ten Raspberry Pi's which perform various functions in my network, including making my house work. I spend all my time just making sure that each Pi is updated and working correctly. I am trying to reduce the required effort by sharing as much as possible between the Pi's.

141
1 + 3
apt
Raffa avatar
jp flag
Raffa
"Are there any apt gurus" ... APT gurus are abundant and will come here eventually ... "I suspect that I am way outside the scope of what apt was designed for." ... Yes, you are ... "The hacker in me is pondering attacking /var/lib/dpkg/* with a Perl script" ... It will attack back and defeat you ... "therefore appear not to exist anymore. So apt kindly tries to re-install them" ... APT doesn't look around ... It only consults its own database, please see my answer to what might be waiting at the end of your current path: [dpkg -l shows only a few installed...](https://askubuntu.com/q/1414284)
0
Reply
Raffa avatar
jp flag
Raffa
BTW `apt` is a front-end for `dpkg`
0
Reply
Stephen Winnall avatar
cn flag
Stephen Winnall
@Raffa Thanks for your feedback. I was aware that `apt` is a front-end for `dpkg`. I promise not to attack `/var/lib/dpkg/*` with a Perl script. I have also upvoted your linked answer, which has helped my understanding a lot. I shall add some further details to the question above, based on my now slightly more informed view of things.
0
Reply
Score:1
Stephen Winnall avatar Ubuntu
cn flag
Stephen Winnall

I'm going to try to answer this myself. Please feel free to comment and point out any errors or false assumptions that I make.

I have enhanced the original question to provide background that might make this answer easier to understand. The question is basically how to use apt to maintain (update, upgrade, install, modify etc.) software packages in ogres, i.e. bootable images for diskless clients, where the images are organised in layers of a union filesystem. Because my NAS only supports aufs, this is the union filesystem that I use in this answer. The methodology should work for other union filesystems too, however.

General principles

  1. Each layer includes a uniquely named file called /etc/ogre-d/<name>.pkg which contains an xargs-compatible list of packages installed by that layer. Whenever a package is added or deleted in a layer the file in /etc/ogre-d/<name>.pkg needs to updated accordingly. This is a manual step. Note that this file only contains the names of packages installed in the current layer (not, for example, those installed in lower layers). Because apt knows nothing about layers it tends to keep lists of all the packages installed in the current layer as well as those installed in lower layers, so it is difficult to derive the packages belonging to the current layer from /var/lib/dpkg/{info, status}.

  2. Layers are updated by running apt upgrade or apt install on each layer individually, starting with the lowest layer and moving incrementally upwards. /etc/ogre-d/<name>.pkg needs to be updated manually in each layer as you proceed.

  3. Since the lowest layer has, by definition, no even lower layer its maintainance with apt is essentially the same as for any conventional Ubuntu installation.

  4. The layers above the lowest layer require special treatment because the apt database (specifically /var/lib/dpkg/{info, status}) needs to be reconstructed from the apt database in its immediately lower layer, and the packages of the current layer need to be added to that. The steps are as follows:

    • on the NAS: delete /var/lib/dpkg/status and /var/lib/dpkg/info in the current layer (this needs to happen in the native filesystem and not in the union filesystem because the unit filesystem would otherwise mark /var/lib/dpkg/status and /var/lib/dpkg/info as deleted and mask them out in the current layer and above)
    • in the ogre: execute DEBIAN_FRONTEND=noninteractive cat /etc/ogre.d/<name>.pkg | xargs apt reinstall -y to restore this layer’s current packages to its apt database (this recreates /var/lib/dpkg/{info, status} from its lower layer, and adds the current packages too)
    • in the ogre: perform any other upgrades or installations, which will also update /var/lib/dpkg/{info, status} in the current layer accordingly
    • proceed to the next higher level and repeat.

  5. When the ogre has been completely updated (from lowest to uppermost), its union filesystem (if it is aufs, in any case) needs to be remounted to ensure the sanity of the uppermost merged directory, which would otherwise contain old versions of files from lower layers that have been modified or deleted in the meantime. On the NAS: mount -t aufs -a -o remount may be sufficient.

Problems to look out for

Some operations (especially delete) need to be carried out on the native directory in which the file is stored on the NAS, because of the .wh mechanism of aufs (lower files, once blanked out, are irretrievable, and we need them to reconstruct /var/lib/dpkg/{info, status}).

On @Raffa's scale of [neat, messy, dirty] I would claim that this solution is neat-to-messy. Please feel free to disagree. If ogres catch on it would be worth providing tools to support the above steps.

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.