Score:2

Server

Extend /var partition on Centos Stream 8

Rafik

12/15/23, 11:48 AM

During the installation of my centos stream 8 OS I gave /var a size of 10GB thinking it would be more than enough, but after I started using docker I found of that it takes too much space on the /var partition as it is shown by the df command:

[root@compute-07 ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs              28G     0   28G   0% /dev
tmpfs                 28G     0   28G   0% /dev/shm
tmpfs                 28G  179M   28G   1% /run
tmpfs                 28G     0   28G   0% /sys/fs/cgroup
/dev/mapper/cs-root  371G  8.2G  363G   3% /
/dev/mapper/cs-home  100G  2.0G   98G   2% /home
/dev/mapper/cs-var    10G  9.0G  1.1G  90% /var
/dev/sda2            2.0G  412M  1.6G  21% /boot
/dev/sda1            2.0G  7.3M  2.0G   1% /boot/efi
/dev/mapper/cs-tmp    10G  105M  9.9G   2% /tmp
tmpfs                5.5G   16K  5.5G   1% /run/user/42
overlay               10G  9.0G  1.1G  90% 

/var/lib/docker/overlay2/77f74478297ca61595f0003d35c7323ec627adb44d94cef92c2b4a3c97319a66/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/5eeb399ce4a3c0d8065d63123985269a359fa66f4d20a8486326e466a48a0128/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/08d42ca66e3a5974bc305b502bca2aa4d22aa7ddcab6ea14c6af300b3abb3a70/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/1a45cb32b127fdf8f5c1483385139a865fdc69260ec20f3dcd4c56ad6890f909/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/ebb9af8edee824a4013dcd526e33e22272283bdee508fd43208fade557def728/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/41d55822108d04c85f348dc134fe6929b0e67f6b3bd7e0af147633bfd3a252c1/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/5cf3e8e06e9b4c2e6ed65164d91aed19f63d55e91524d6563e9f16b6709d29be/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/1d1eda94e68596b00d0cabcc75a7c91999a1c845ebd02c733bb9f00ac66a26f0/merged
overlay               10G  9.0G  1.1G  90% /var/lib/docker/overlay2/bc9e9a28c6761ad89a9457110642156ebd2b4d77de841f2a3ac9ffbaa90e4b63/merged

tmpfs                5.5G     0  5.5G   0% /run/user/0

This is the output of lsblk command, as /var is in a separate parition then / and /home I couldn't find a way to resize it or create another partition then link it to /var

[root@compute-07 ~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 558.4G  0 disk 
├─sda1        8:1    0     2G  0 part /boot/efi
├─sda2        8:2    0     2G  0 part /boot
└─sda3        8:3    0 554.4G  0 part 
  ├─cs-root 253:0    0 370.4G  0 lvm  /
  ├─cs-swap 253:1    0    64G  0 lvm  [SWAP]
  ├─cs-tmp  253:2    0    10G  0 lvm  /tmp
  ├─cs-home 253:3    0   100G  0 lvm  /home
  └─cs-var  253:4    0    10G  0 lvm  /var
sr0          11:0    1  1024M  0 rom

Is there any way I could resize this partition without having to lose any data.

606

3 + 1

partition

filesystems

lvm

centos

docker

Criggie

12/16/23, 2:33 AM

Does `vgdisplay` show any "Free PE" in the Physical Volume? Is this a virtual machine where you can just add an additional disk easily for /var/lib/docker ?

0

Reply

Score:6

Server

Nikita Kipriyanov

12/15/23, 12:12 PM

No. Actually, most of the space in the system is always used by /var where normally variable data lives — databases, Docker, logs, and so on, while Linux's /, rarely uses more than 10 GiB of space, when the system is properly managed.

Not only your LVM is suboptimal: I'd fit the whole Debian installation with some services into the space you wasted in your ESP and /boot. (I had a habit of creating VMs with 4GiB allocated for the system. More than enough.)

So well, the best solution would be to reduce your oversized root file system, but that couldn't be done online. You need to boot some rescue media, activate your VG and then reduce the file system and the logical volume and that could consume substantial time, considering the size of the partition. Also it is possible to mess things. If you're in hurry, you can make a twist: make some directory in root, move the Docker data into it and bind-mount it back into where Docker data live in var. You need to stop Docker and all containers for the time being. Something like this:

systemctl stop docker.service
mkdir /var-lib-docker
mv /var/lib/docker/* /var-lib-docker
mount --bind /var-lib-docker /var/lib/docker
systemctl start docker.service

That way, the Docker will use the space on the oversized root file system, while being available at the usual location. In the /etc/fstab you need to add the following entry, so this configuration will survive reboot:

/var-lib-docker /var/lib/docker none bind 0 0

But remember this is quite ugly hack which can buy you some time but also bring some problems, so look for the opportunity to reclaim space from the root and allocating it to /var properly.

And let this be the lesson: never allocate all avaliable space from the beginning. Leave some (most) of it unallocated. You can always easily add the space and do that online, while it is very hard, time consuming and error prone to reclaim it, and that requires downtime. It is best to manage the space so you never need to reclaim it.

+ 4

Rafik

12/15/23, 12:20 PM

Is it too risky to resize the root partition? or should I just reinstall the system from scratch if I can ?

0

Reply

Nikita Kipriyanov

12/15/23, 12:24 PM

Not riskier than any other partititon. I'd say data is always more valuable than the system, especially when you are ready to reinstall it. It likely will be faster than reinstall, though. Depending on the file system you used, it could be more or less time consuming (e.g. `xfs` was desingned with "no reclaim" in mind, so while it is possible, it is ill suited). And you can always back up those 8.2 GiB somewhere, just for the case.

2

Reply

Rafik

12/15/23, 12:27 PM

and what would be the recommended allocated space for /var partition in my case? thank you for your answers

0

Reply

Nikita Kipriyanov

12/15/23, 12:31 PM

I'd make both root and var 20 GiB each. I repeat this, you will be able to enlarge all of them online, without any downtime and impact. Just monitor the system and add the space in a timely manner, when, say, more that 75% is full or less than 5-10 GiB free space left. (var usage *will* grow, root is unlikely to grow)

2

Reply

Score:3

Server

diya

12/15/23, 12:34 PM

As an alternative to this answer

Your system is using LVM logical volumes.

There may be some unused space left in the volume group that you can assign to extend var.

In case there isn't any unused space left, or too little, you can reduce or remove the swap LVM and take the space that gets freed up by that to extend /var.

That would leave your system intact and can typically even be done online.

The approach for that would be something along the lines of (commands untested, so don't copy them outright but the principle should be sound)

Create a temporary or permanent swap file on your oversized root file system as replacement while you take the swap partition offline.
- dd if=/dev/zero of=/swap-file count=65536 bs=1M
- mkswap /swap-file
- swapon /swap-file
swapoff /dev/mapper/cs-swap to disable the swap partition
lvreduce cs-swap followed by formatting the remaining LVM swap partition again with mkswap and re-activating the smaller swap space with swapon
or delete it ouright with lvremove

Use the newly created free LVM space to extend the /var LVM volume and extend the file system system

lvextend -l +100%FREE --resizefs /dev/mapper/cs-var

+ 2

Nikita Kipriyanov

12/15/23, 12:55 PM

Whoa, the swap is oversized too. Good catch. I didn't notice it.

1

Reply

diya

12/15/23, 1:47 PM

I figure that reclaiming the 64 G swap would be good work-around

1

Reply

Score:1

Server

iBug

12/16/23, 6:34 AM

Your df and lsblk don't tell us what their filesystems are. To shrink a volume (partition or LV), the filesystem on it must be shrunk first. A common choice for rootfs, ext4, can be shrunk offline. The default FS choice for CentOS (and RHEL-like distros), however, is XFS, and cannot be shrunk at present.

As others have pointed out in their answers, your rootfs usage 8.2 GB is not difficult to back up. So depending on your rootfs filesystem type, you have the following options:

If it's ext4, you can shrink it in-place:

Reboot to another system, like an ArchISO, as ext4 can only be shrunk offline.
Make sure lvs shows your partition, then e2fsck -f /dev/mapper/cs-root
resize2fs -M -p /dev/mapper/cs-root
- Alternatively, if you know your desired rootfs size, you can specify it instead of using -M:
  
  resize2fs -p /dev/mapper/cs-root 15G
  
  Note that it's a better idea to leave some space for shrinking the underlying volume.
Reclaim space from the LV lvreduce -L 16G cs/root
Resize the filesystem back to the volume size: resize2fs -p /dev/mapper/cs-root
Now the free space can be handed out to /var: lvextend -l +100%FREE cs/var

If it's XFS, however, your best bet is to backup your rootfs and recreate it:

Boot up an ArchISO like above.
rsync -avHAXx /your-rootfs /somewhere (Make sure /somewhere is on another volume - it's easy to find 8.2 GB of spare space - your swap volume is very likely oversized)
lvremove the root volume and recreate it with an appropriate size. I recommend formatting to ext4 this time.
rsync -avHAXx /somewhere /your-rootfs to copy things back.
Update your GRUB configuration if needed - AFAICT the LV ID is often written into grub.cfg as a hint, but I'm not familiar with CentOS (on Debian it'd be simply update-grub).
Assign free space in the VG as above.

Other notes I'd like to point out:

Your EFI system partition (ESP) is too large. For a typical Linux server, 256 MB should be more than enough (I use 128 MB on my Ubuntu server). It only needs to hold the GRUB binary and a very basic version of grub.cfg.
/boot doesn't have to live on its own volume. GRUB can read LVM and your rootfs filesystem just fine.
Depending on your server design, your /root is likely oversized as well (I use 16 GB for my Proxmox VE server - I'm not planning to put data on the host).
Do you really need 64 GB of swap space? Normally I'm fine with just 8 GB.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Extend /var partition on Centos Stream 8

Extend /var partition on Centos Stream 8

Post an answer