How dangerous might it be - and what performance gains may be had - by turning vulnerability mitigations off on non-Internet facing servers?

When a virtual machine Linux host server is non-Internet facing and is used exclusively on a LAN and is using a relatively well tested distribution like Proxmox, how dangerous would it be to turn off all vulnerability mitigations via the kernel arg mitigations=off?

Additionally, has anyone tested what kinds of performance gains might be seen by turning off all such mitigations?

This recently came to be a question for me when I saw this big hit that the retbleed mitigations create: https://www.phoronix.com/review/retbleed-benchmark

This line of thought extended to the curiosity of what may be the ramifications - both ill and positive - of removing all or some mitigations either via the above kernel argument or by individually turning off high-impact mitigations.

The Linux kernel flag mitigations=[on|off] is a single toggle to easily enable/disable all available kernel mitigations for hardware vulnerabilities as listed here https://docs.kernel.org/admin-guide/hw-vuln/index.html

The impact of that depends of course entirely on your CPU:

• When your CPU isn’t vulnerable to any of the known vulnerabilities, then none of the mitigations are applicable and the impact should effectively be zero.

• When your CPU is vulnerable to some (are there even CPUs vulnerable to all of them?) the impact depends on which specific vulnerabilities and your workload.

As to a risk analysis, that also depends on your workload and user base.

On a virtualization host operated by public VPS provider, the guests are trusted less and much more likely to be malicious (or compromised) than I would expect on a internal virtualization host used exclusively by my colleagues.

For example on our virtualization hosts used for CI/CD pipelines and compute clusters, all the guests are short lived, get deployed from trusted images, run for up to a couple of hours and then get destroyed again. There we need all the performance we can get and disable the mitigations.

On a different shared cluster we host more classical server consolidation workloads; guests that get deployed there there can (and are more likely to) run “forever” rather than for hours. It has a mix of production and non-production workloads and the guests are managed by DevOps teams that are not all as diligent in patching and updating their systems and applications.
There the risk of a malicious or compromised guest is much higher, the possible reduced performance for specific workloads an acceptable trade-off and thus mitigations do get enabled there and we limit which CPU flags get exposed to the guests.

is the server really impossible to connect to even indirectly from outside your network?

That's the first question you should ask yourself. If the LAN can be accessed from a machine that has 2 network connections, one to that LAN and another to another network that is connected to the internet (hopefully through several layers of firewalls) you just connected that server to the internet.

And remember that a machine doesn't need to be connected to the internet for it to be vulnerable to attacks. A machine could be infected with malware that is simply intent on doing damage to it via a USB stick, floppy disk, or even typed in commands on a terminal.

In the end, all security, as all performance improvements, is a compromise. What's an acceptable level of risk for the rewards gained.

To find out your potential performance gains, run tests on the machine while it has no network connections at all and see for yourself. Most likely it won't be very much, but it might just be enough to squeeze out those few extra CPU cycles that help some old hardware survive just that bit longer that you have the time to convince upper management that you really, really, should get the budget for a new server.