Score:2

ESX: FS on Datastore corrupted after volume on storage was FULL

in flag

I observed this failure : FS on Datastore corrupted after volume on storage was FULL, actually it happened twice, and only option was to recreate again new datastore.

Here is what happened in time order:

  • datastore disconnected (re-exported to ESXi hosts)
  • volume on storage array was FULL on 100%
  • FS on Datastore corrupted (data lost )

Do you have any idea what can be root cause ? Unfortunately I have no logs. Only clue is that DS was always full before FS crashed. Is this standard behavior ? We are using multiple ESX versions connected to this storage : ESX ver. 7, 6.5

Score:4
ru flag

If you use thin provisioning for VMDKs, and especially with the ESXi swapping/logging on the same volume then you must never run out of storage. If a thin-provisioned VMDK cannot grow, the VM must be frozen - freezing also requires storage, so the VM necessarily suffers damage.

At the same time, the host cannot write to its own vital storage and begins freezing. There's really no failure mode out of this any more.

An ESXi would need to detect that danger a while before it really happens, bringing with it quite a few intricacies (data may be generated/written quite quickly). So, it's generally your responsibility as the admin: Make sure that you monitor disk usage closely, that you migrate growing VMs to other storage early on, and that you are ready to add more disks for storage expansion.

andrew avatar
in flag
yep, this is probably the root cause, Thanks for info !
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.