To summarize:
I want to run multiple servers on a host, for different groups of users, and allow them to add and modify functionality of the servers at runtime with some form of scripting. This is a 100% necessary feature for what I want to do, but there is obviously a lot of potential for this to go horribly wrong; what are some actions/approaches I can take to mitigate the risks involved?
It's probably worth mentioning that I am not a system administrator and am most likely unaware of many best practices taken for granted by those that are; my ideas are based on what seems logical to me given what little I know - if any of them seem misguided or if I'm missing some important ideas, please assume I am ignorant of what should be done and let me know.
To elaborate:
Users will connect to a master server, which will handle authentication and facilitate accessing their scriptable servers of choice. I'm expecting that users will establish a connection to a scriptable server and afterward communicate directly to it for a session; that seems logical from the point of view of an application developer, I'm not sure if it's a bad security practice?
All the servers will run linux and be run on a linux host.
The master server will have a database, probably PostgreSQL, with some user-related data e.g. for authentication. It will probably also have some data regarding the scriptable servers so it can advertise them and handle connecting users to them.
The scriptable servers will need some user-related data as well, but mostly will contain user-created content; each scriptable server will store that data in an SQLite database.
Users should not be able to run any code on the master server.
On the scriptable servers, I envision them using a language like Tcl or Lua, which can be embedded and allow exposing only part of their functionality to users. Tcl does this via a "safe interpreter"; Lua apparently has some sandboxing capability.
I don't expect these language features to completely protect my servers from being compromised.
I am considering running all the servers in their own rootless docker containers. I know this is not enough to contain a compromise, but my understanding is that it should help.
The next part is where I am especially unsure what to do:
Running each server in its own virtual machine with VirtualBox or the like would provide what I imagine would be very good isolation, however it would also consume a lot of resources; I need to avoid that, or else find a linux distro that only consumes something like 5-20 MB of RAM, which seems unlikely to me and would still likely be way more resources than the servers themselves would use, which would be very unappealing.
Is there a good answer or alternative to this?
Ultimately, while I hope to avoid any system being compromised, I expect it to happen at some point or another; if somebody wants to do that, they're going to figure it out.
I'm looking for a resource-efficient way to make it difficult for a single compromised server to lead to others being compromised; even better would be if I could keep the compromised server from being used to wreak havoc across the Internet. I'd also like to have a good way to recover from this; I intend to take backups and send them elsewhere, but I'd be interested in any other suggestions - I could see that being another question, however.
