GPU Acceleration on a Windows Server without virtualization over RDP

I'm trying to find out if it's possible to run a Windows Server with one GPU which is shared between all RDP clients so that people could

• create a session on the server
• start some program with a UI which needs GPU acceleration
• disconnect afterwards while the program stays running and gets full acceleration
• later reconnect to the session

Maybe that's an unusual use case because most things i can find about Windows Server and GPU seem to be about virtualization, f.e. here where it's even mentioned that

if your workload runs directly on physical Windows Server hosts, then you have no need for graphics virtualization; your apps and services already have access to the GPU capabilities and APIs natively supported in Windows Server

which might indicate that is is possible.

I've read about RemoteFX and GPU-Partitioning, f.e. here, but it again looks like this is only for virtualization and i don't care about how fast rdp would update remote screens as long as the running programs get the full acceleration.

Am i searching for the wrong things? Is this even possible?

If it's possible, how would it impact performance when the session is connected and when it's disconnected?

As it's a physical server you need to instruct your server to use it's own GPU for the RDP client that connect to it.

It's there;

Local Computer Policy\Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Remote Session Environment

Then enable “Use the hardware default graphics adapter for all Remote Desktop Services sessions”

A printscreen done, sorry my OS is in French, but it's the location.

Please note the OS of the users that connect must be minimum in Windows 10 too.

The limit you can hit is more the GPU memory if your application is not intensive on the GPU. It would be the calculate how much users can use the application before the video ram is depleted.