Immutable Linux images - how far can and should I go?

Immutability is cool in order to realize consistency, predictability and reliability, and I don’t see any reason why I shouldn’t strive for OS-level immutability when deploying my application on Linux VPS’es from different cloud providers around the world. With tools like Packer to help create OS images, this also seems to be the way to go.

For some cloud providers (e.g. Digital Ocean), I can build the image locally in qcow2 or raw format and then upload the completed image to the cloud provider for deployment when instantiating new VPS’es. This seems to be the best option.

But other cloud providers (e.g. Hetzner in Germany) don’t support importing your own OS images - instead you have to build the image on their infrastructure based on one of their source images, and then you can snapshot the final installation into a reusable image, when everything is configured correctly. This in fact is also, what the “Hetzner Cloud Builder” from Packer does.

But how do I then guarantee, that the final Hetzner image has the exact same CentOS 8 installation (down to the precise same set of installed RPM packes with the exact same version numbers) as is running on all other cloud providers?

I imagine that the solution could be some kind of declarative tool, which takes a list of RPM packes and associated version numbers and brings the target system in line with this list - ensuring that any missing RPM packages are installed in the right version, removing superfluous RPM packes, upgrading older RPM packages and downgrading newer RPM packages to ensure that the required version is installed.

Does such a tool exist, or should I think completely differently about this?

Some might argue, that the CentOS RPM packages should always just be upgraded to the newest available version, but then I can not guarantee that all cloud providers are running the same OS installation - which potentially can affect predictability and reliability of my service.

Instead, I want to be able to thoroughly test a complete setup (OS + application) before deploying it to any cloud provider, and then the deployment shall be the same across all cloud providers. This is how we do things at the application level using Docker images, and I can’t see why we should accept any less at the OS level.

Any input from you fellow DevSecOps colleagues how to reach these goals?

But how do I then guarantee, that the final Hetzner image has the exact same CentOS 8 installation (down to the precise same set of installed RPM packes with the exact same version numbers) as is running on all other cloud providers?

If I understand correctly what you want, I would go to any 'infra-test' tool like Inspec which lets you describe what you want to have in your target image/VM.

We use it to validate our saltstack/saltproject code using it in conjunction with Kitchen this way (we run it in a job on our CI tool)

• create N virtual machines with Kitchen (can be used with Docker, Vagrant but also cloud providers)
• provision each machine with its profile, with Salt, Ansible, Chef, Puppet or whatever
• verify the resulting machine state with Inspec

Inspec lets you create sort of 'unit tests' but for infrastructure/system: you can easily verify users and groups presents or absents, packages installed and their version, services running, TCP/UDP ports, firewall rules...

(I also use Packer to create images but at the moment I don't use Inspec in this context: we consider that the code used to create this image has already been tested by the previous CI job)

So, going to your problem: I would add to your Packer setup an Inspec verification step. Moreover it seems that it's already integrated in Packer as a provisioner https://www.packer.io/docs/provisioners/inspec (which I just discovered)

In fact, once you've created VM in Hetzner, you can boot it to a LiveCD of choice, which has things like CloneZilla or SystemRescue which you can use to dump/restore your image.

I actually find that CloneZilla images are more portable between various cloud VPS providers.

Neither image based updates nor immutable hosts are required to set some controls over what software is installed. Options exist between the extremes of
auto updating packages from public mirrors and immutable images.

With a distro using individual packages, say CentOS using rpm, could maintain your own updates repo. Limit to just the packages you use. Fetch updates into a testing repo regularly, say every 4 weeks. Test this frozen set of packages for a while, then move into your stable repo. Configure all hosts to automatically update from your mirror, on a schedule perhaps. When automation installs a package as part of deploying a thing, its version is known because it comes from your mirror. However, individual hosts may have problems applying the package transaction, and fall out of date. Consider querying every host in the fleet for versions of particularly important packages, perhaps certain security updates.

Easy enough to clone an OS install as another instance. However, images of package based distros are not really immutable, the package manager is still available in the instance, and privilaged users can still change things in the /usr tree.

Consider a distro built around image based updating, like CoreOS. Updates must be layered into the image, and rebooted to take effect. Customization is limited to a small amount of metadata. CoreOS in particular is only for hosting containers, but this may be your use case. Has the advantage of being reproducible, Fedora CoreOS 34.20210904.3.0 is a well defined set of software.

Compute hosts that do not provide a way to upload images can be worked around. Boot a rescue environment with network access, and download the image directly onto the disk. Snapshot that to make a template.

Now might be a good time to evaluate your choice of distro for other reasons. CentOS Linux 8 ends December 2021. CentOS Stream is the replacement, however it is upstream rather than downstream from RHEL.