Join Log in
Score:6
Niall Gallagher avatar Server

AWS instance giving me less NVMe space than advertised

br flag
Niall Gallagher

I recently provisioned 10 m5d.large instances (with Terrafor. The advertised space is "1 x 75 NVMe SSD", however when I ssh on to the instance I see.

[root@web0 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.8G     0  3.8G   0% /dev/shm
tmpfs           3.8G  376K  3.8G   1% /run
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   30G  2.5G   27G   9% /
tmpfs           777M     0  777M   0% /run/user/1000
tmpfs           777M     0  777M   0% /run/user/0

I also see in the AWS console 10 Elastic Block volumes of 30 GB.

Why am I not getting the 75 GB advertised https://aws.amazon.com/ec2/instance-types/m5/?

Update

Running fdisk shows the same

[ec2-user@web0 ~]$ sudo fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 324AAE3D-30C7-46AB-9185-9756BFACB090

Device           Start      End  Sectors Size Type
/dev/nvme0n1p1    4096 62914526 62910431  30G Linux filesystem
/dev/nvme0n1p128  2048     4095     2048   1M BIOS boot

Partition table entries are not in disk order.

Update

Running lsblk does show something

[ec2-user@web0 ~]$ lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0 69.9G  0 disk
nvme0n1       259:1    0   30G  0 disk
├─nvme0n1p1   259:2    0   30G  0 part /
└─nvme0n1p128 259:3    0    1M  0 part

Not sure what is going on here, this is a vanilla provision with Terraform, perhaps I just need to mount the 69 GB disk. Can't see where this 30 GB EBS is coming from or why its mounted on /.

2497
2 + 3
Michael Hampton avatar
cz flag
Michael Hampton
Did you happen to ask explicitly for 30GB disks in your terraform configuration?
0
Reply
Niall Gallagher avatar
br flag
Niall Gallagher
I did not specifically request it. When doing a ```terraform plan``` I see that the root_block_device is "(known after apply)". When I dump out the JSON (created from apply) I see ```"throughput": 0, "volume_size": 30, "volume_type": "gp2"```. The same block also shows "ebs_block_device": [].
0
Reply
Niall Gallagher avatar
br flag
Niall Gallagher
Shot in the dark here, but https://aws.amazon.com/ebs/pricing/ says 30 GB is in the "free tier", I wonder if this is just a complementary EBS volume? If it is, ill bet there are many deployments/customers using it over the attached NVMe disk.
0
Reply
Score:15
MLu avatar Server
id flag
MLu

You've got 2 disks as lsblk confirms:

  • /dev/nvme0n1 is your EBS volume with the operating system, created from the AMI. Most likely the 30GB size is the default size prescribed by the AMI you used.

    It's not free though - you pay ca $0.10/GB/month (depending on the region and storage type)

    This is automatically mounted and used by your instance.

  • /dev/nvme1n1 is your Ephemeral storage (or Instance storage). It's faster, cheaper, but will be wiped when the instance is stopped (by you or if it crashes).

    It comes unpartitioned and unformatted and unmounted - you have to do that all yourself in your startup scripts.

    Only use it for data that you don't mind losing! E.g. cache, or something that can be recreated easily.

More info about Instance storage

Hope that helps :)

0 + 3
joshudson avatar
cn flag
joshudson
/dev/nvme1n1 sounds a good place to put a swap file.
0
Reply
MLu avatar
id flag
MLu
@joshudson Indeed, if you need 75GB of swap space… ;)
0
Reply
joshudson avatar
cn flag
joshudson
I mount `/tmp` as swapfs and compile stuff there.
0
Reply
Score:5
avatar Server
kr flag
dyasny

Looks like you provisioned a 30gb ebs root disk, that's the one mounted on / and you have the added ephemeral nvme1n1 showing as well.

Format nvme1n1 and mount it wherever you need it. Keep in mind the data on such a disk will not survive a cold boot of the instance.

0 + 5
Niall Gallagher avatar
br flag
Niall Gallagher
Thing is I did not provision the 30 GB EBS. It just appeared. I have no problem keeping it if its free, however I find it odd that it exists at all.
0
Reply
Tim avatar
gp flag
Tim
I haven't used nvme instances, but AFAIK your OS disk is always going to be EBS as it's durable. If you don't explicitly provision the EBS disk it's probably automatic.
2
Reply
Niall Gallagher avatar
br flag
Niall Gallagher
This is what I am thinking also.
0
Reply
kr flag
dyasny
Exactly. Depending on the AMI there's a default root disk size provisioned, and the root disk AFAIK always comes from ebs
0
Reply
Criggie avatar
in flag
Criggie
Correct - there is no good way to use ephemeral disk as a boot device. It is wiped on a stop-start, while "probably" surviving a plain reboot which leaves the instance inside the same AWS droplet.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.