Score:0

GCP Compute Engine full disk not resized file system on public image

us flag

GCP VM with 99,8% full disk has not resized its file system after increasing the disk capacity in google console.

I have a small VM on GCP based on public image ubuntu-2004-focal-v20220419. There was a 10GB disk with root partition and file system. Some logs occupied 99,8% of the disk capacity. I am able ssh the VM because GCP can still copy ssh keys to it, but to detect a heavy folder I had to use another disk to keep temporary files:

sudo du -Sh | sort -rh -T /dev/tmp | head -5

instead of

sudo du -hs * | sort -rh | head -10

I increased the disk size to 15GB (without removing any data) and restarted the VM. Documentation says:

For VMs with public images, Compute Engine automatically resizes the root partition and file system after you increase the size of the boot disk and restart the VM.

however, I can see that file system was not resized

~$ sudo lsblk
sda       8:0    0    15G  0 disk 
├─sda1    8:1    0   9.9G  0 part /
├─sda14   8:14   0     4M  0 part 
└─sda15   8:15   0   106M  0 part /boot/efi

Right after restarting the VM I can se following message in Logs Explorer:

{
  insertId: "1"
  jsonPayload: {
    @type: "type.googleapis.com/cloud_integrity.IntegrityEvent"
    bootCounter: "13"
    earlyBootReportEvent: {
      actualMeasurements: [8]
      policyEvaluationPassed: false
      policyMeasurements: [3]
    }
  }
  logName: "projects/<vm name>/logs/compute.googleapis.com%2Fshielded_vm_integrity"
  receiveTimestamp: "2022-11-30T09:12:33.141683678Z"
  resource: {2}
  severity: "ERROR"
  timestamp: "2022-11-30T09:12:32.752150154Z"
}

I followed steps on jhanley.com and there are no entries regarding resizing. There are also neither expand-root.sh script nor expand-root service.

I am wondering why is it so? I expected that additional 5GBs will be merged with sda1 automatically based on above documentation. Could it be, that the disk is so full, so some GCP background processes are not able to resize the file system?

[EDIT] I created a snapshot of the disk and resized file share manually. In the serial port output I can now see:

Nov 30 11:55:30 <vm name> kernel: [    9.016905] EXT4-fs (sda1): resizing filesystem from 4165883 to 4428027 blocks
Nov 30 11:55:30 <vm name> kernel: [    9.021130] EXT4-fs (sda1): resized filesystem to 4428027
かいぜん avatar
pm flag
Tried replicating this on my end using the public image `ubuntu-2004-focal-v20220419` after using the `fallocate` command to fill up my 10G disk to 99% then follow the same set of instruction from [Increase the size of a disk](https://cloud.google.com/compute/docs/disks/resize-persistent-disk#increase_the_size_of_a_disk) and setting the size of the disk to 15G the disk resize successfully and right now I have a free space of 66% on my disk, I advice you to try creating a new VM with the same specs and test if it will have the same behavior after resizing your disk.
かいぜん avatar
pm flag
see also this documentation on [Troubleshooting full disks and disk resizing.](https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-disk-full-resize)
John Hanley avatar
cn flag
You have posted this question twice. On your other post, I showed you how to read the OS startup logs in my article. Those logs will tell you why the resize did not succeed. What do the logs tell you? From your last edit you now have two problems to analyze.
J.Wincewicz avatar
us flag
@JohnHanley thank you for suggestion. Thanks to your blog post I was able to prove that resizing was not run by system itself - what happened after resizing file system and expanding disk size later on
かいぜん avatar
pm flag
@J.Wincewicz for greater visibility of the community can you provide how did you resolve your issue.
Score:1
us flag

Finally I was able to fix it. In my case disk was too full what prevented system from running resizing file system scripts.

After reading John Hanley blog post I realized that file system resizing has never happened. I resized file system and partitions myself as described here:

sudo parted /dev/sda

and then, I extended the file system:

sudo resize2fs /dev/sda1

That should have been done automatically, what normally happens, but again, due to disk clogging mostly, the process was not able to be performed. After manual resizing of file system and resizing the disk capacity in GCP I see that now it works as it is suppose to:

Nov 30 11:55:30 <vm name> kernel: [    9.016905] EXT4-fs (sda1): resizing filesystem from 4165883 to 4428027 blocks
Nov 30 11:55:30 <vm name> kernel: [    9.021130] EXT4-fs (sda1): resized filesystem to 4428027
> lsblk
sda       8:0    0    17G  0 disk 
├─sda1    8:1    0  16.9G  0 part /
├─sda14   8:14   0     4M  0 part 
└─sda15   8:15   0   106M  0 part /boot/efi
> sudo df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4       17G   11G  6.2G  63% /
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.