Score:4

How can I make automatic, full system backups of Ubuntu during normal use (like Macrium Reflect for Windows)

in flag

I want to be able to make backups of my Ubuntu install automatically, during normal use. It's important that this is an automatic process because I'm not going to remember to regularly boot into another environment to make a backup. This rules out most of the things I see suggested, like using dd, or CloneZilla, or gnome-disks.

I'm currently using the Déjà Dup Backup utility that comes with Ubuntu 20.04 to back up my files automatically, but if my hard drive fails I'm afraid I'll be doing a lot of work to get my environment set back up again.

I'm open to creative solutions that give me automatic full-system backups, even if they involve booting into another environment, as long as they happen automatically during my normal daily use.

For example, running my main OS in a virtual machine, and automating snapshots and backups of those VM files. I've seen this suggested offhand in a comment, but don't know how realistic a solution it is.


Edit:

  • My main concern is hard drive failure. I need to be able to restore onto a new drive.

  • All I use the device for are personal coding projects. I don't do any hosting or anything from it.

  • It's important to me that all of my installed software and configuration is backed up, not just my files.

  • In the title I mentioned Macrium Reflect. The features I find desirable from that software is that it runs on the same drive that it's backing up, automatically, and makes an image that can be completely restored even onto another drive.

    Ideally, I'd like to back up my system such that after restoring onto another drive, I could type localhost/currentproject into my address bar, and have my current project load. Or open my IDE and see all of my project files. I want it to feel as though my drive never failed.

ChanganAuto avatar
us flag
Cloning and backup aren't the same thing. Although a cloned drive image can be used and understood as backup it is indeed impractical for what you rightfully so want, i.e., incremental and versioned. There are specific tools for that that aren't related to cloning. I'm confident the answers will soon provide you with the actual solutions you're looking for.
Paul avatar
us flag
I been using Duplicity as part of Mail-in-a-Box, which configures it automagically, but high on my list is installing to my other servers. It does incremental encrypted backups and can be configured to sync to remote servers, including S3 API compatible services, as well as standard `rscync` and I think others.
in flag
Umm … ZFS can do all of this for you. Automatic snapshots, replication across storage devices, flawless restoration of the file system to a specific point in time
Obscerno avatar
in flag
@matigo ZFS looks very interesting, I've never heard of it before now. It looks like I'd have to start over on a new install if I wanted to use this as my root file system, though it looks pretty easy to set up since it's included in the Ubuntu installer. I'm going to have to do more reading on this to grok it, but if you want to write up an answer on how you would use ZFS to back up a drive and then later restore from it, I know I'd appreciate it! Thanks regardless for sending me down this path
Obscerno avatar
in flag
So it looks like I'd use `zfs snapshot` on my main install, then use `zfs send` with that snapshot to send it to the other drive, and then `zfs recieve` to restore from the other drive. I'm not ready to commit to ZFS, but thought I'd throw these commands out there in case anyone stumbles here from google and wants a signpost to follow.
Score:3
in flag

Update: this solution messes up snaps due to a limitation in Timeshift. I've added a workaround to the steps for now, until this is fixed.


This is meant as complementary to Organic Marble's answer.

So you've decided to use Timeshift to make a full backup of your system. Excellent! It's good software. But there are some things that you need to be aware of.

But before I get into it, this needs to be said up front: Make sure you try restoring a backup before your drive fails. Restoring to a new drive is not as straightforward as you think.

Things to keep in mind as you set up Timeshift

1. If you want to back up user home directories, you have to enable that manually

To do this:

  • In Timeshift's settings, go to the "Users" tab and select "Include All Files".
  • In the "Filters" tab, click the "+" for each of the default filters, as by default they exclude /home files.

Only do this if you want Timeshift to back up your personal files as well as your system. This isn't the intended use-case for Timeshift, but since this question is about recovering everything to a new drive, I thought I should include this. And it works very well.

2. After you've set up Timeshift, save your disk structure somewhere

The disk structure is important to the restore process. Timeshift doesn't handle creating partitions or anything; you have to add those partitions yourself, and then tell Timeshift to use them. So pull up your system drive in GParted and email a screenshot to yourself or store it somewhere you'll be able to get to it if your drive fails.

Here's an example from my system. As you can see, my installation is split across two partitions:

My system loaded in GParted. There are two partitions. The first partition - /dev/sda1 - is named "EFI System Partition". It's formatted as fat32 and takes up 512MiB. It has the flags "boot" and "esp. The second partition - /dev/sda2 - has no name. It's formatted as ext4, and takes up the remaining space on the drive. It has no flags.

How to restore to a new drive

1. Create a bootable Ubuntu USB stick and load into the desktop

This is covered elsewhere. There's a tutorial here.

We're going to use this to restore the snapshot to the new drive. Boot into the USB, and select the "Try" options until you're into the desktop.

2. Install needed tools

GParted

Already installed!

Timeshift

sudo apt-get update
sudo apt-get install timeshift

Boot-Repair

sudo add-apt-repository ppa:yannubuntu/boot-repair && sudo apt update
sudo apt install -y boot-repair && boot-repair

3. Prepare the target drive

You want your target drive to match the structure that your system drive had. So use your reference to create the needed partitions in GParted, in the correct format, with the correct flags:

The target drive, loaded in GParted after being modified to match the structure of my original system. There are two partitions. The first partition - /dev/sdb1 - is named "EFI System Partition". It's formatted as fat32 and takes up 512MiB. It has the flags "boot" and "esp. The second partition - /dev/sdb2 - has no name. It's formatted as ext4, and takes up the remaining space on the drive. It has no flags.

4. Use Timeshift to restore to the target drive

First, open Timeshift, and when prompted to "Select Snapshot Location", select the device that you were backing up your system to (i.e. the one that you want to restore from):

The Timeshift "Select Snapshot Location" prompt. A drive, "Maxone" is selected.

Your Timeshift snapshots should pop up. Select the one you want to restore onto your target drive, and select "Restore". You will get a popup that looks like this:

The Timeshift "Select Target Device" prompt. The root drive is set to sdb2, and the /boot/efi drive is set to sdb1. The other paths are set to "Keep on Root Device"

Important: I had to manually set the /boot/efi to use the first partition of the target drive. Timeshift doesn't do this for you manually. Your system might not have the same structure as mine, so you may have to do something different here. That's why it is so important that you test your backup and make sure it can be restored, because you may need to reference your currently working system to see how it's structured.

After this, continue with the restore process. When it's done, we have one thing left to do!

5. Use Boot-Repair to make the drive bootable

This may not be necessary for you, but it absolutely was for me. Launch Boot-Repair and select "Recommended Repair". You may want to look at the advanced options to make sure it's detecting your drive correctly. For me, it did, so the recommended repair worked fine.

The Boot-Repair dialogue. It has "Recommended Repair" button, a "Create BootInfo Summary" button, and an "Advanced options" dropdown.

6. Boot into the target drive to make sure the backup worked

If you're loading it on the same device that your original system is on, you should change the desktop image or something to make sure you don't boot into the existing system and think your backup worked! Did this happen to me? Maybe.

For me, these steps completely restored my computer. The desktop environment and all of my settings and projects were left intact.

If this didn't work for you, be glad you're trying it now and not finding out after your drive failed that you don't know how to restore your backup! I spent a lot of time struggling with this, and if you have issues I might not be able to help. But hopefully this helps steer you in the right direction!

7. UPDATE: Fix snaps

Due to a current bug in Timeshift, snaps are broken when restoring to a new drive. To fix this:

  • Use this method to repair each of the snap cores. Short version: run snap refresh core --edge then snap refresh core --stable to force refresh the core. I did this for core, core18, core20, and core22, just to be safe.
  • Remove then install the rest of the apps. You can see them all with snap list. The user data as far as I can tell is retained and it also backs it all up on removal anyway, so you can restore if your data is lost (see snap saved after removing an app).

Other notes

Timeshift uses symlinks to back up your system, which can be a bit of a pain if you want to make a remote backup. Backblaze, for example, doesn't support symlinks at all.

I've found that .tar files work well for packaging up the backup into a format that can be uploaded, and I was able to successfully restore from an extracted .tar file.

I haven't settled on .tar yet, but if you decide to do something similar, make sure to test that the backup works after extracting! Some formats strip symlinks by default which will completely break your backup.

PrakashS avatar
mx flag
Though step 5 is not necessary as specified, you may better re-install grub in chroot environment.
Score:3
us flag

Timeshift is pretty powerful and is included in the Ubuntu universe repo.

Plusses:

  • It runs in the background at a frequency you specify.

  • It actually works ...I had been doing Timeshift backups for years, but had never done a restore until last week. It restored my system perfectly...but kind of reset my desktop environment...but that is what Deja-Dup is for. Between the two of them, it seems to cover most cases.

Minuses:

  • You can't back up over the network. Backing up to the physical drive your system is on is kind of dumb...so this is OK for multi-drive systems but not so good for, e.g., laptops. For small single disk desktop systems I stick a 64G USB in the back and set Timeshift to back up to that.
Obscerno avatar
in flag
Timeshift combined with Déjà Dup sounds promising, but I have some questions. Most important one first, can it restore onto another drive? My main concern is drive failure (I'll edit to make that more explicit in my question). Reading the linked page, I get the impression that Timeshift is designed to rewind a system back to a usable state, and I'm not sure it would work to use the restore on a different system.
Organic Marble avatar
us flag
@Obscerno In fact, what I just used it for was to swap out a failing hard drive. When the drive failed, I had a Clonezilla clone that was a couple of months old. I restored that clone to a new hard drive, booted into it, and used Timeshift to bring it up to date. So I personally haven't used Timeshift alone to do it, but I see no reason why it wouldn't work.
Obscerno avatar
in flag
Thanks! I'll have to investigate that, but if in the end I need to make one Clonezilla clone that's fine, as long as I don't have to continuously make clones. So then if Timeshift can restore system files and settings, what do I need to back up with Déjà Dup to make sure I've covered everything? Also, will this also cover installed packages or will I need to do something like this as well: https://askubuntu.com/a/99151/1118096
Organic Marble avatar
us flag
Backing up a list of installed programs is kind of redundant if you do Timeshift - but full confession, I do it myself. The reason I used the Clonezilla clone is that I knew it would boot and then I could "Timeshift" back to where I was. The documents say you can boot to a live USB, install Timeshift there in the live system, and do a full system restore that way, but I just had never done it, and I was on Plan D by that point anyway. I just back up /home/[myuser] with Deja-dup.
Obscerno avatar
in flag
Thank you so much! I'm going to wait a bit to see if any other answers roll in but this is my current plan. I was a bit confused by the language in the docs. They say "If the main system is not bootable, then it is possible to boot from an Ubuntu Live CD, install Timeshift on the live system, and restore a snapshot on the main system." which sounds like you use the Live CD to restore to the original drive. But they say you can restore after installing a different distro which suggests it should work on a fresh install? I need to get my hands on another drive so I can test it out.
Obscerno avatar
in flag
Just a follow up on this, I'm finally getting around to doing a test restore on another drive, and I'm having a hell of a time getting it bootable. From what I can tell, Timeshift isn't setting up partitions or anything. That said, I saw there is an option to put /boot on a separate partition, so maybe I have to manually add partitions and then do that? Still, my hope was that someone less technical could do this if I got hit by a bus. But maybe I'm just doing it wrong... I'll keep tinkering.
Obscerno avatar
in flag
Follow up #2: I did it! I did indeed have to create the partitions manually, and when I restored, I needed to specify the /boot/efi partition. I also had to run a boot-repair. But in the end it worked perfectly! It remembered everything including my desktop icons. (I backed up the /home directory). I'll make a complement answer with the details on restoring soon. I'm going to do the whole thing over again and get pictures to illustrate with this time too.
Organic Marble avatar
us flag
@Obscerno sounds like that could be really helpful.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.