how to automate servers installation with virtualization/LAMP (small IaC)?

How to automate server installation? What infrastructure/systems fit our needs?

We have over 10 phisical servers with over 100 VMs, mostly LAMP. Not too many but also not anything that can keep working by itself. All servers have a KVM virtualization. We host various custom made applications for our customers that we modify frequently. So this is not a single application that needs to scale. Project teams work directly with customers and with VMs and this is a requirement.

The problem we routinely get into is that reinstallation of a VM/machine gets tricky since nobody controls recent changes to them, it poses a risk of instability, people defere it as long as possible and the problems get bigger over time.

I would like to establish a reinstallation procedure at least every 3 months and on demand. Reinstallation should start with a code repository and a backup. This should not be a VM duplication.

We estimate that this will cost us about 6 manmonths every 3 months in the current setup if done by hand. How can we reduce this time and also increase repetability of the process?

The question is what software can help us with automating this task. It should be as lightweight as possible. We do not need automated node allocation. We just need to have reinstallations to be as automated as possible. We assume human oversight of each reinstallation.

Our nodes are mostly LAMP with two systems having over 1TB structured DB data, several >200GB and around 50 small, custom VMs. We can schedule a planned downtime of each system easily, so this is not a question about high availability during reinstallation.

We just need to automate reinstallation as much as possible.

Does your cloud have an API key?

Most clouds have one-click LAMP images, e.g.

Vultr: https://www.vultr.com/apps/lamp/

Digital Ocean: https://marketplace.digitalocean.com/apps/lamp

Linode: https://www.linode.com/docs/guides/lamp-stack-marketplace-app/

If not, my absolute go-to is VestaCP https://vestacp.com/install/

And https://hestiacp.com/

Both add apache backend, nginx front-end, MySQL/mariadb/postgres, and php/php-fpm. All optionals too.

For updates, I would create a separate server, drop a common SSH public key in /root/.ssh/authorized_keys that the separated server has, which can automatically log into each server, something like:

INSTANCES=(
10.0.0.5
10.0.0.6
10.0.0.7
...
)


for INSTANCE in "${INSTANCES[@]}"; do
    ssh root@${INSTANCE} /bin/bash -c "apt update -y && apt upgrade -y"
done

This is pretty much Terraform, Kubernetes etc. DIY.

Also on GitHub, there's loads of cloud-specific guides, for example:

Another Digital Ocean one: https://github.com/ethanbeyer/DigitalOcean-Droplet-Setup

An AWS one: https://github.com/elionaz/aws-LAMP/blob/master/install_lamp.sh