Production AWS Instance: We have an m5 ec2 nitro instance that is in need of query optimization. However, we are in a time crunch and have decided to temporarily throw memory and storage at this until we can complete the optimization of queries. (Apologies in advance)
There is 50GB on one EBS volume (/dev/sda1) and another 50GB on a second volume (/dev/sdf). As far as I can tell, the EC2 is only using one of the 50GB volumes, dev/sda1.
Linux Kernal version: 4.4.0-1128-aws
32GB RAM (M5a.2xlarge)
Our /dev/nvme0n1p1 | (/tmp) directory is filling up creating an SQL 28 No Disk Space error and we want to increase the size of the /tmp directory from 20GB to 50GB. while we optimize queries to reduce the sizes of the temporary database (.MAI) files stored in /tmp
Where I am at: In our testing environment/EC2 (also an M5.2xlarge) I was able to increase the volume size to 100GB and I have followed the steps in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
- This is what shows when I run lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 100G 0 disk
├─nvme0n1p1 259:1 0 20G 0 part /
├─nvme0n1p2 259:2 0 2G 0 part [SWAP]
└─nvme0n1p3 259:3 0 28G 0 part
├─vg_xxx-logs 251:0 0 8G 0 lvm /var/log
└─vg_xxx-app 251:1 0 19G 0 lvm /home/xxx
This is what shows when I run df -hT:
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 16G 0 16G 0% /dev
tmpfs tmpfs 3.1G 183M 3.0G 6% /run
/dev/nvme0n1p1 ext4 20G 8.6G 11G 45% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 squashfs 44M 44M 0 100% /snap/certbot/1788
/dev/loop1 squashfs 111M 111M 0 100% /snap/core/12834
/dev/loop3 squashfs 62M 62M 0 100% /snap/core20/1434
/dev/loop5 squashfs 56M 56M 0 100% /snap/core18/2409
/dev/loop4 squashfs 25M 25M 0 100% /snap/amazon-ssm-agent/4046
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/2284
/dev/mapper/vg_xxx-logs xfs 8.0G 362M 7.7G 5% /var/log
/dev/loop6 squashfs 26M 26M 0 100% /snap/amazon-ssm-agent/5656
/dev/loop8 squashfs 44M 44M 0 100% /snap/certbot/2035
/dev/loop7 squashfs 62M 62M 0 100% /snap/core20/1328
/dev/mapper/vg_xxx-app xfs 19G 4.7G 15G 25% /home/xxx
tmpfs tmpfs 3.1G 0 3.1G 0% /run/user/1000
tmpfs tmpfs 3.1G 0 3.1G 0% /run/user/1001
As you can see, it shows that nvme0n1 has 100GB available, however, the 3 partitions still equal 50GB. when I get to step 7 on the AWS documentation on extending the filesystem to occupy the new added volume space, I get the following:
ubuntu@ip-xx-xx-xx-xxx:~$ **sudo resize2fs /dev/nvme0n1p1**
resize2fs 1.42.13 (17-May-2015)
**The filesystem is already 5242619 (4k) blocks long. Nothing to do!**
I have an ext4 filesystem (except that I see two lvms in the sub directory of nvme0n1p3 but I don’t think that changes anything) and have tried growpart, parted, but these solutions found online are generally for Ubuntu and not specifically for EC2 EBS volumes, so I do not want to divert too much away from what should be a AWS provided solution that I cannot seem to find. That being said, those also produced error messages saying the drive/directory was in use.
My understanding is that AWS EBS allows for an ec2 to increase volume size and then extend the file-system without stopping the instance or unmounting the device being extended. Nonetheless, I have not been able to do the same when stopping the staging instance I am testing on. I am open to stopping the instance to get this done, however, would prefer to do it without unmounting device and/or stopping instance.
I can also upgrade the instance to provide more RAM if need be, but I’d still need to extend the filesystem first.
Any help is appreciated!
