Score:-2

A Complete Guide For Transition From Single to Multiple Partition Ubuntu 20.04 Server On Already Installed System Using LVM

tr flag

The title is my request. I have been reading for hours and I have found tutorials for moving /home to a partition and /tmp to a partition. But none for converting all recommended directories into partitions for Ubuntu Server. One tutorial got close, but then ignored the obviously necessary step of copying data from prior directories to new partitions.

Anyway, after many hours, I still have not yet found a tutorial that covers my rather common situation.

Situation:
I installed Ubuntu Server on Prod & Test servers (vps & virtualbox, respectively), did some work. While reading server hardening tutorials, I realized that it is best practice to have /boot /home /swapfile /tmp /usr and var... /opt? all on separate partitions for resource management and security reasons.

The correct way to create the above mentioned partitions is by using LVM

However, none of tutorials discuss / recommend partition sizes (proportionally, as obviously different systems have different disk space available, mine is 80GB on / )

Additionally, some tutorials mention the necessary changes to /etc/fstab and some don't.

My goal is to convert my 80GB single partition setup into multiple partitions and then secure the partitions against common attacks and exploits using: a combination of nodev,noexec,nosuid in /etc/fstab

I can spend days testing and failing different configurations in virtualbox... Or some handsome and/or beautiful "Ubuntu Server Partitioning Guru" can publish an easy to follow (intermediate user target audience) definitive guide as I have described above and become Internet Famous as many users hosting their own projects on vps will forever love & adore you. :-)

UPDATE
I originally did not publish the primary server hardening guide becuse I didnt want this post to become a "debate" over this guy's very thorough piece of work. However, in hindsight I think it would be helpful for folks looking to answer this post, to see the actual security benefits that I am trying to achieve.

https://linoxide.com/ultimate-guide-secure-ubuntu/

in flag
Are you trying to use different *partitions* or different *volumes*? While they may appear the same from 30,000 feet, they operate a bit differently inside a 2U rack mount. As you are trying to move core directories, the easiest way to do this would be with an Ubuntu Desktop Live Session, which will give you full control over the Ubuntu Server storage devices ... but I don't see a whole lot of gain here. As someone who has used Ubuntu Server for well over a decade in all sorts of situations, this sort of setup has never been necessary. LVs for data? Yes. LVs for everything? No ...
nightwatch avatar
tr flag
@matigo difference volume and partition is definitely new to guess, I will have to read more. Based your experience, which directories should be on different "partitions" given my limited understanding? Only /tmp and /home? and just not worry about the others?
in flag
Without getting too far into "opinions", the only things that generally need their own volume/partition on Ubuntu Server is the data that is most accessed/important. If it's a database/domain/mail server, then the DB should be on a really fast device and the logs on a separate device. If it's a web server, then it really doesn't matter because the whole system is disposable. I generally view the core OS itself as irrelevant, as it's able to be rebuilt from scratch in under an hour (worst case) and is just fine with the default setup. Only the data (and logs) matter ... (for *my* fleet)
nightwatch avatar
tr flag
I understand your rationale and logic. So i should focus my energy on moving /home (my data / public_html directory) and /var/log to separate partitions as you are correct with vps. I can restore the system itself in minutes. Any recommended reading / tutorials to achieve this?
in flag
[This answer from 10 years ago](https://askubuntu.com/a/50539/1222991) is still valid today. As you're working with some VMs as well, be sure to practise on those instances first if you're worried about taking a production system down with an erroneous step. `/var/log` only needs to be elsewhere if you are recording *a lot* of data (20MB/sec or more), which is very rare. If you're just running a web server, having your code in GitHub (or some other repo) and the uploaded data backed up and SSH'd off the system will be sufficient without separate volumes/partitions.
in flag
If your primary goal is to protect data against various forms of online attacks, the biggest favour you can do yourself is to use AppArmor and the Uncomplicated Firewall (ufw). These two tools alone will block 99.9% of the stupid crap that losers will throw at your server. If you are running a WordPress site with a plugin that gets pWn3d, then AppArmor will ensure Apache cannot write outside `/var/www`, so the base OS will be mostly safe so long as you're not messing around with system permission structures.
nightwatch avatar
tr flag
Thanks for the point in the right direction. I have read enough web server hardening guides to get dizzy. With regards to firewalls.. I am running Virtualmin with Firewalld and I have a VPS firewall provided by my ISP, this should be sufficient in lieu of UFW.. (Please correct me if I am wrong.)
nightwatch avatar
tr flag
@matigo thx for letting pick your brain. I will put my paranoia to rest and get back to developing websites. Enjoy your weekend.
Score:2
eg flag

I think usually separating things into different file systems doesn't help/improve security. If someone breaks into the system while it's running, anyway everything is mounted and there is no logical difference to having everything in one file system. What reasons were given in the material you studied?

That said, it can help performance (different file systems or different hardware beneath these mount points) and shorten how long a disaster recovery takes (e.g. if only the SSD that held /var went up in smoke, you only have to restore that backup and the rest stays running).

You were asking about recommended partition sizes:

  • For / you're good with 15-20 GB. I've never needed more for a server (running a web server + mail server).
  • swap I usually put on its own volume / partition which has the same size as the RAM - simply so that suspend-to-disk can work. People used to recommend 2xRAM size, but with nowadays' RAM sizes, when you run into a situation where you need a serious amount of swap, you're anyway in trouble. You'll notice it from a massive slowdown, and you should then quickly increase the available RAM.
  • /boot just needs 500 MB, which gives it space for 8+ kernel+initrd versions. Make sure to run apt autoremove frequently, to keep it trimmed after kernel upgrades.
  • I would keep /usr and /opt as directories on /, just don't see a benefit in moving them to their own filesystems.
  • As described in this hardening guide, having a separate volume for /tmp does make sense, as it allows you to make that world-writable directory more restricted - both in how much space it can use of the totally available space, and what can be done with the files that are stored there. The guide recommends to use the nodev, nosuid and noexec options when mounting the filesystem. The guide only gives the mount commands for "one-time use". Translating this into a line in /etc/fstab would mean that you place nodev,nosuid,noexec into the 4th (options) column of the line where you mount the dedicated /tmp volume onto /tmp.
  • That leaves us with /home and /var - that's usually the "important" stuff. On my servers /home is pretty much empty, but /var holds public_html, the logs, the databases, etc. So I keep /home as a directory on /, but /var definitely gets its own volume, and is backed up most frequently. Give it all remaining space after the above was done.

Then you asked how to make the transition:

  • In the running system, where everything is on /, attach the new disk (VDI file, ...), prepare it with pvcreate, lvcreate and then the file systems of your choice (mkfs.ext4 for example).
  • Then make temporary mount points under /mnt, e.g. /mnt/newroot, /mnt/newvar, ... and mount the file systems there.
  • Then use rsync -xaP <source>/ <destination>/ for each of your file systems. The '-x' option will prevent rsync from crossing file system boundaries, i.e. if you do rsync -xaP / /mnt/newroot/ it won't also copy /var, /home or even all the new filesystems mounted under /mnt. '-a' will make sure permissions etc. will be taken over without modification, and '-P' shows progress. For details, please refer to man rsync.

After that's done, edit the /mnt/newroot/etc/fstab and make sure you list all file systems at the appropriate mount points. If you've gotten that far, this shouldn't be too hard (as you chose all the /dev/mapper/... names, file systems, etc.).

You will also have to use grub-install and maybe update-grub to make the new disk bootable, but there I'm not so sure about the exact procedure. With VMs you can easily try it out and if it doesn't boot, attach the old disk again and fix it.

For reference, here is a shell session that gives you some specific commands on the partitioning + LVM + formatting + referencing in fstab topics. Please note that you most likely will have to modify them, for example if your device isn't /dev/sda, if you want different file systems, etc. - it's merely an example.

# After using fdisk to create one partition that covers the whole device,
# it looks like this:
root@ubuntu:~# fdisk -l /dev/sda
Disk /dev/sda: 100 GiB, 107374182400 bytes, 209715200 sectors
Disk model: VBOX HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 18ACB4C9-3F33-7041-8BEB-D819F138A809

Device     Start       End   Sectors  Size Type
/dev/sda1   2048 209715166 209713119  100G Linux LVM

# Create a physical volume for LVM
root@ubuntu:~# pvcreate /dev/sda1   
  Physical volume "/dev/sda1" successfully created.
  
# Create a volume group with the name "vg1" for LVM that will
# hold all our logical volumes
root@ubuntu:~# vgcreate vg1 /dev/sda1
  Volume group "vg1" successfully created

# Create the logical volumes as described above
root@ubuntu:~# lvcreate --name root --size 20G vg1     
  Logical volume "root" created.
root@ubuntu:~# lvcreate --name swap --size 8G vg1
  Logical volume "swap" created.
root@ubuntu:~# lvcreate --name boot --size 500M vg1
  Logical volume "boot" created.
root@ubuntu:~# lvcreate --name tmp --size 5G vg1
  Logical volume "tmp" created.

# Have a look at the logical volumes
root@ubuntu:~# lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  boot vg1 -wi-a----- 500.00m                                                    
  root vg1 -wi-a-----  20.00g                                                    
  swap vg1 -wi-a-----   8.00g                                                    
  tmp  vg1 -wi-a-----   5.00g                                                    

# Have a look at the volume group and see how much space is left
root@ubuntu:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree  
  vg1   1   4   0 wz--n- <100.00g <66.51g

# Use the remaining space for the last logical volume, var
root@ubuntu:~# lvcreate --name var --size 66.5G vg1
  Logical volume "var" created.

# Have another look at the volumes
root@ubuntu:~# lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  boot vg1 -wi-a----- 500.00m                                                    
  root vg1 -wi-a-----  20.00g                                                    
  swap vg1 -wi-a-----   8.00g                                                    
  tmp  vg1 -wi-a-----   5.00g                                                    
  var  vg1 -wi-a-----  66.50g

# Format all volumes with ext4 file system
for i in /dev/mapper/vg1-*; do mkfs.ext4 $i; done
# Turn vg1-swap into swap space
mkswap /dev/mapper/vg1-swap

# Create fstab entries that look like this
/dev/mapper/vg1-root    /       ext4    defaults                0       1
/dev/mapper/vg1-boot    /boot   ext4    defaults                0       2
/dev/mapper/vg1-var     /var    ext4    defaults                0       2
/dev/mapper/vg1-tmp     /tmp    ext4    nosuid,nodev,noexec     0       0
/dev/mapper/vg1-swap    none    swap    sw                      0       0
nightwatch avatar
tr flag
this of one of many server hardening guides I've been following. Note the security measures for fstab that have been repeated in other places. https://linoxide.com/ultimate-guide-secure-ubuntu/
eg flag
Thanks. The recommendations indeed make sense, especially the resource exhaustion parts. I hadn't considered that. Not sure what happens to the services that use /tmp when it's full though (even when it's on another volume). Maybe they will anyway stop working and thus you will still suffer a DoS.
nightwatch avatar
tr flag
might you be willing to update your answer based upon the hardening guide?
eg flag
@nightwatch done :)
nightwatch avatar
tr flag
I highly appreciate the reply and I have already up voted the response as it is useful and I learned from it. However, I was hoping for practical examples of creating the partitions using LVM as recommended in the guide. From various tutorials I know that creating a /tmp partition has some difference from creating for /home. Those fine points could cost me hours.
eg flag
@nightwatch I've added a shell session where I did the partitioning + LVM + formatting + referencing in fstab and added it to the answer, but it's still just an example. In your specific case it might look different. I think you would reach your goal faster if you asked specific questions when you get stuck at one point.
nightwatch avatar
tr flag
"for President?" you have my vote :-) thanks
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.