swappiness does not force use of swap space. Nor will it save you from not having enough memory.
Higher values of swappiness encourages reclaim of anonymous pages, not just page cache. But this does not do much for ZFS on Linux, which does not use Linux's page cache.
I basically want there to always be a certain amount of RAM free. ... To
add my system current is using 230/256GB of RAM with no swap being
used yet. ... if I create another virtual machine when the RAM is at 99%
it wont boot.
Do some capacity planning to not oversubscribe memory. Less of a magic command to tell the hypervisor to keep free memory around, and more your discipline of not starting more guests than you have resources for.
Your 230/256GB is 90% utilized, much higher than this can get into memory pressure, not good for performance. Which may call for capping guest memory, 56 x 4 GB guests, to make up some numbers. Whether the couple dozen GB remaining is enough to run the hypervisor kernel and still have some reserve is something you can discover in testing.
Edit: From meminfo, your 500 GB host is under some memory pressure and is swapping out.
- MemAvailable at 5.8% of total is low. 29 GB to work with on a 500 GB host is not very much.
- SwapTotal minus SwapFree shows 285 GB of swap space use. 1788 GB of total swap means it is not going to run out any time soon. Remember that most persistent storage is orders of magnitude slower than DRAM.
- 0.4 GB Cached is quite low in absolute numbers. Consistent with ZFS on Linux use which isn't using the usual Linux VFS page cache. As a result, swappiness tunable does almost nothing in this environment. If you are dropping caches manually, don't do that, it likely hurts performance.
Swapping out is in sets of pages at a time when needed. The host will not suddenly free up an entire 100 GB guest when guest memory demands are lower. That would be very expensive.
I am skeptical of memory oversubscription in general, and ballooning in particular, and do not recommend them. Encouraging low memory can be risky to performance, as in the worst case reclaim introduces latency and might anger the OOM killer. See your attempts to start guests at high utilization, beyond a certain point the kernel won't grant the memory allocations.
Confirm the host has > 100 GB RAM (not counting swap) available before starting a 100 GB guest. Shut down guests before reducing their memory size. Not oversubscribing is more expensive in memory costs, but has more consistent performance and is easier to maintain.
