How to setup recovery on Ubuntu server?

We have Ubuntu server on the remote location that may or may not have an Internet access. It will run several applications and we want to make sure that if something goes wrong on OS or app level, we can automatically recover to the previous working version.

First we were thinking to go with Timeshift, but after some testing it doesn't feel like it is robust enough.

Now we are thinking to make an additional partition, where we will store a copy of our primary one using dd command. If something goes wrong we will just restore from that.

Is there a better alternative or known best practice for this scenario?

My first thought was to run the ubuntu server on another ubuntu hypervisor. Keep a clone of the VM offline and simply boot it when required.

All of it comes out of the box and is even very simple to script. If you have a way of checking if your service is working you could simply use the virsh interface to shutdown and bootup the spare VM.

If you need even more redundancy you could keep this spare VM on a different server (in case of disk failure etc)

Virt manager is a little more friendly than the CLI interface and can be used remotely too! https://virt-manager.org/

The CLI interface is quite simple too, to boot up vm called guest1

virsh start guest1

Free, simple and reliable. Also no scary dd hacks :)

Probably a million ways to do it but this is what I would go for.

Considering how many time I have installed Linux or reinstalled Linux on a server, I would probably take the Ansible route NEXT time I create a server.

That is:

• First make a image containing the minimal server that you can configure remotely via Ansible.

• Then make an Ansible playbook that contains all the services (aka "ansible roles") and settings you need.

From here on you basically just need to backup any user generated files like websites, mails, databases and the like.

And the backup and restore process can also be done as an Ansible playbook.

It goes without saying: Keep a backup of your Ansible playbooks! :-)

The advantage of using Ansible is that besides using it to backup and restore the server, it can also be used to spin up additional servers, should the need arise for additional capacity.

And maintaining one vs many servers in Ansible is relatively trivial, because you can group servers together that fits the same role in your inventory list.

Any changes applied to the group will be rolled out to all servers in the group.

Too long for a comment:

Generally your back-up and restore procedures, recovery and/or roll-backs will be needed after the following failure scenario's (in no particular order):

• failed upgrades/changes
• operator error
• hardware failure(s)
• server compromise ( malicious actions , malware, virus etc)

When your (only) backups are stored on the same server with the original data they won't protect against and can't be relied on for most of those failure scenarios.

A common approach is that you safeguard unique data with a schedule of full and incremental backups (see for example https://en.wikipedia.org/wiki/Backup and https://en.wikipedia.org/wiki/List_of_backup_software) and have an automated deployment and centralised configuration management as means to quickly re-deploy a failed server.