SAN Solution for diverse VM Servers

I recently got my fingers into Server Administration. I am currently the only person administering an air gapped network with 8 sites. Most things are dated and old but I get the money to renew things.

Currently we run a couple of Servers with ESXi and Proxmox at each site. All new Servers are ESXi and all old ones Proxmox. All of these Servers have Local SSD Storage on the Box. Additionally we have 2 TrueNAS Enterprise Servers with a lot of HDD in 2 Different locations, currently connected via 1G Ethernet Links. These are capable of NFS and iSCSI and all this, but are too slow to keep the Storage of the VMs, even in the same Site. All sites have at least one OS2 Cable link to 2 other Sites currently running 1G L3 links. The onsite Networks are mostly 10G/40G as they are already renewed by me. 2 Sites have 4x40G L3 links to each other for redundancy reasons, this will be extended to 3 Sites.

As we have very high demand on availability because we run a good portion of OT hardware and PLCs, local storage on the server is not a good solution for me. Management promises some ridiculously high SLAs on some Servers and hardware I currently can't held up to.

I can backup the VMs on the Truenas but it would certainly feel better to not have the machines located on the Compute units anyway. Things like my DNS Server are at risk of failing if one Node fails. If this happens, I want to be able to boot it on another node and lose a couple of minutes and not all the time since the last backup which is a lot of time as I don't have much time to backup regularly.

As I am not a Storage Person I'm looking for Solutions which are easy to maintain and fast enough for the local VMs. We don't run any Windows Servers, solutions containing windows are also not feasible as I don't know how to maintain them air gapped.

Now the real Questions.

Which is a good solution for local onsite SSD storage SAN or NAS?

Should we just get 2 or 3 other TrueNAS with SSD and depend on that or are we missing opportunities by doing so? This doesn't feel very enterprise, but the service for the current HDD Truenas was very good.

Is there anything wrong with going 40G NFS with jumbo frames or should I build something different like iSCSI?

What is the real advantage of a block based storage in 2022?

I'm a little bit lost in the middle of VMware sales looking into selling vSAN or other people who want to deploy netapp for everything everywhere because they see € in their eyes.

I don't want to build a unicorn when I don't get much out of it reliability wise, but also don't want to miss out on opportunities to get more redundancy or other pros out of the storage.

I hope This is an appropriate question for this audience and hope you can help me out. I'm happy to get more data if it is necessary.

Thank you in advance!

nay reason not to use hyperconverged? If you need HA you need 3 servers per location anyway. Yes, 3 - not 2. One is in patching, one can fail at that time.

This is enough to use Hyperconverged setup. 100 gigabit backbone - grab the Ethernet cards from Nvidia (Mellanox) and use Mikrotik 100G router (2 of them, again) for the backbone, laugh all the way to the bank.

NOT a ESX person but my development cluster uses 3 of those using Windows Server. There is NO downtime for the VM's even when doing stuff like hardware upgrades. No external storage at all - right now we use 1u servers wiht 2 SSD for cache and 8 SATA HDD. The next upgrade (our' are ZEN1 EPYP) will see that likely replaced with 2U cabinets for 24 discs and LIKELY a full SSD move. Stupid 2.5" capacity is really not growing at all it seems, or (like the new 5TB HDD) getting SLOWER. Someone please make a 15mm 4TB PMR hdd with nearline quality and SATA interface? I replace them all immediately.

The performance impact of a hyperconverged ROCE using 100G backbone is something not to sneer at. Whoever had the 1G idea was out of his mind - even 10 years ago I would have gone with 10g on an Intel 520 card back then. We actually did.

And this setup is fully redundant out of the box. With serious overhead, but you cannot avoid having 2 reserve servers. Gets better should you grow - still only need 2 reserve - but then you need WAY more expensive switches than Mikrotik. They sadly only produce 4 port at the moment for 100G.And anyone saying "hah, how cheap can they be" - below 1000 USD. 908 actually, last time I checked. Compare that to prices of big brand 100g and you get shocked.

An external NAS is really only needed if you need VERY excessive storage. And even then I would go with something like Windows S2D - real distributed storage.

I would recommend Pure Storage, the offer an all-flash array, I have never seen faster storage in my 15-year IT engineering career. And once good implemented it's very stable and realiable. For backup Veeam Backup is the standard in the VMware world.