Limiting ARC Size for the proposed system

I have a pool of 5 x enterprise 12Gps SSD that I use for internal storage for a proxmox server. This pool is used to stand up VMs and containers. The problem is that it is eating up a lot of my available RAM which is 192G! Here is the summary:

As you can see the Max size is 94.4GiB!!!

My question is: What should be the optimal value for this? and how to change it?

This is my production server and I want to be 100% sure of the process. (I already looked here but my server is also not honouring the arc size :( )

Any help or suggestions would be highly appreciated!

You can persistently reduce the ARC size by doing the following.

Edit or add a file named

/etc/modprobe.d/zfs.conf

Add a line as follows. This example line is for 1 gig of memory. So multiply the number by the number of gigs you want it to be sized.

options zfs zfs_arc_max=1073741824

After saving the file run the command

update-initramfs -u -k all

Then reboot to apply the changes.

By default, ARC uses up to 50% of system RAM. However, ARC is dynamic: it will tune itself down if the system experience increased memory pressure.

So I suggest capping it down only if you experience some real system issue, or in system with really large RAM where having so much cache is of no real advantage.

That said, you can manually (and dynamically) cap ARC via the zfs_arc_max tunable. However, be sure to understand that simply reducing it below current ARC usage will not immediately release memory. From the man page:

zfs_arc_max can be changed dynamically with some caveats. It cannot be set back to 0 while running and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking.

If you need to immediately free ARC memory, you can set the tunable above and then manually dropping cache via echo 3 > /proc/sys/vm/drop_caches.

Alternatively, you can permanently set it via /etc/modprobe.d/zfs.conf and reboot your system.