Score:0

Extending File-system after EBS volume increase on M5 type EC2 with Ubuntu 16.04

in flag

Production AWS Instance: We have an m5 ec2 nitro instance that is in need of query optimization. However, we are in a time crunch and have decided to temporarily throw memory and storage at this until we can complete the optimization of queries. (Apologies in advance)

There is 50GB on one EBS volume (/dev/sda1) and another 50GB on a second volume (/dev/sdf). As far as I can tell, the EC2 is only using one of the 50GB volumes, dev/sda1.

Linux Kernal version: 4.4.0-1128-aws 32GB RAM (M5a.2xlarge)

Our /dev/nvme0n1p1 | (/tmp) directory is filling up creating an SQL 28 No Disk Space error and we want to increase the size of the /tmp directory from 20GB to 50GB. while we optimize queries to reduce the sizes of the temporary database (.MAI) files stored in /tmp

Where I am at: In our testing environment/EC2 (also an M5.2xlarge) I was able to increase the volume size to 100GB and I have followed the steps in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html

  • This is what shows when I run lsblk
NAME                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

nvme0n1             259:0    0   100G  0 disk                                                                                                         
├─nvme0n1p1         259:1    0    20G  0 part /                                                                                                       
├─nvme0n1p2         259:2    0     2G  0 part [SWAP]                                                                                                  
└─nvme0n1p3         259:3    0    28G  0 part                                                                                                         
  ├─vg_xxx-logs     251:0    0     8G  0 lvm  /var/log                                                                                                
  └─vg_xxx-app      251:1    0    19G  0 lvm  /home/xxx

This is what shows when I run df -hT:

Filesystem              Type      Size  Used Avail Use% Mounted on
udev                    devtmpfs   16G     0   16G   0% /dev
tmpfs                   tmpfs     3.1G  183M  3.0G   6% /run
/dev/nvme0n1p1          ext4       20G  8.6G   11G  45% /
tmpfs                   tmpfs      16G     0   16G   0% /dev/shm
tmpfs                   tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                   tmpfs      16G     0   16G   0% /sys/fs/cgroup
/dev/loop0              squashfs   44M   44M     0 100% /snap/certbot/1788
/dev/loop1              squashfs  111M  111M     0 100% /snap/core/12834
/dev/loop3              squashfs   62M   62M     0 100% /snap/core20/1434
/dev/loop5              squashfs   56M   56M     0 100% /snap/core18/2409
/dev/loop4              squashfs   25M   25M     0 100% /snap/amazon-ssm-agent/4046
/dev/loop2              squashfs   56M   56M     0 100% /snap/core18/2284
/dev/mapper/vg_xxx-logs xfs       8.0G  362M  7.7G   5% /var/log
/dev/loop6              squashfs   26M   26M     0 100% /snap/amazon-ssm-agent/5656
/dev/loop8              squashfs   44M   44M     0 100% /snap/certbot/2035
/dev/loop7              squashfs   62M   62M     0 100% /snap/core20/1328
/dev/mapper/vg_xxx-app  xfs        19G  4.7G   15G  25% /home/xxx
tmpfs                   tmpfs     3.1G     0  3.1G   0% /run/user/1000
tmpfs                   tmpfs     3.1G     0  3.1G   0% /run/user/1001

As you can see, it shows that nvme0n1 has 100GB available, however, the 3 partitions still equal 50GB. when I get to step 7 on the AWS documentation on extending the filesystem to occupy the new added volume space, I get the following:

ubuntu@ip-xx-xx-xx-xxx:~$ **sudo resize2fs /dev/nvme0n1p1**                                                                                           
resize2fs 1.42.13 (17-May-2015)                                                                                                                   
**The filesystem is already 5242619 (4k) blocks long.  Nothing to do!**

I have an ext4 filesystem (except that I see two lvms in the sub directory of nvme0n1p3 but I don’t think that changes anything) and have tried growpart, parted, but these solutions found online are generally for Ubuntu and not specifically for EC2 EBS volumes, so I do not want to divert too much away from what should be a AWS provided solution that I cannot seem to find. That being said, those also produced error messages saying the drive/directory was in use.

My understanding is that AWS EBS allows for an ec2 to increase volume size and then extend the file-system without stopping the instance or unmounting the device being extended. Nonetheless, I have not been able to do the same when stopping the staging instance I am testing on. I am open to stopping the instance to get this done, however, would prefer to do it without unmounting device and/or stopping instance.

I can also upgrade the instance to provide more RAM if need be, but I’d still need to extend the filesystem first.

Any help is appreciated!

me flag
you must grow your partition first before you can grow your filesystem. EBS is just a normal disk and nothing special or different.
in flag
Thank you, It wont let me grow the 1st partition, only the 3rd one. Do I need to move the 2nd and 3rd partitions first? Or does EBS handle all of that even if you select a partition that is adjacent to another one or more?
Score:0
cn flag

Extending a Linux file system after resizing a volume:

  1. login to instance;

  2. Type df -h to list the info about the file system;

  3. Type lsblk;

  4. Check your volume name;

  5. To extend the partition on the root volume, use the following growpart command.

    Notice that there is a space between the device name and the partition number.

    growpart /dev/nvme0n1 1
    
    # (1 = number of partitions)
    
  6. Check the size again with df -h and lsblk

  7. After extending the partition, extend the whole filesystem resize2fs /dev/nvme0n1p1

in flag
So I am able to get it to work when I use it on the last partition (/dev/nvme0n1 3) and that partition will grow, however, it will not work when I type the command to resize the first partition (/dev/nvme0n 1). I have seen examples of using GUI programs that can accomplish this, but I need to know how to expand the first partition to allow for the resize to take effect and that would entail moving the 2nd and third partitions, correct?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.