I am trying to deploy an RD Gateway in combination with WAP (Web Application Proxy) and AD FS pre-authentication as described here.
For a "proof of concept", I've decided to deploy all RDS roles to one server. Simplified, my environment now looks something like this:

Where the server labeled "RDS" contains these roles:
- RD Web Access
- RD Gateway
- RD Licensing
- RD Connection Broker
- RD Virtualization Host
On the AD FS Farm, I configured the following Relying Part Trust, which only has the identifier set:

And on the WAP, the published application looks like this:

Now, Internally everything works. A client in DEVPROD can access RD Web and connect to the vdi resources.
On the WAP, everything works. On any server of the farm, I can access RD Web and connect to the vdi resources.
From outside, I can access RD Web, but connections to the RD Gateway fail with this error message:
On some clients, I also get:
Your computer can't connect to the remote computer because the Remote Desktop Gateway server is temporarily unavailable.
What I've tried/checked
- All certs used are trusted and rdweb uses the correct one
- IIS does not have unused bindings
- Using windows authentication for IIS
- Setting pre-authentication to required in the custom rdp properties of the collection
- Setting DefaultTSGateway and radcmserver in the IIS application settings
Where would you start diagnosing this issue?