WinRM connectivity issues with workgroup and vlan subnet

Situation:

1. 1 server (Windows Server 2022) (192.168.15.5)
2. 1 client (Windows 11) (192.168.5.5)
3. Unifi network setup where both machines are on a different VLAN (subnet)

On the server:

1. Run Winrm quickconfig
2. Run Enable-PSRemoting
3. Run Get-NetFirewallRule -Name 'WINRM*' | Get-NetFirewallAddressFilter | Set-NetFirewallAddressFilter -RemoteAddress Any (to make sure other subnet should be able to connect)

On the client:

1. Run Winrm quickconfig
2. Run Enable-PSRemoting
3. Run Set-Item WSMan:\localhost\Client\TrustedHosts –Value "192.168.15.5"

When running test-wsman [server ip], sometimes it works, sometimes it doesn't:

Call 1 (seems to be good):

PS C:\WINDOWS\system32> test-wsman 192.168.15.5


wsmid           : http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd
ProtocolVersion : http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
ProductVendor   : Microsoft Corporation
ProductVersion  : OS: 0.0.0 SP: 0.0 Stack: 3.0

Call 2 (done a few seconds later):

PS C:\WINDOWS\system32> test-wsman 192.168.15.5
test-wsman : <f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2150859046" Machine="WO
RKSTATION"><f:Message>WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. By default, the WinRM firewall exception for public profiles limits access to remote computers within the same local subnet. </f:Message></f:WSManFault>
At line:1 char:1
+ test-wsman 192.168.15.5
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (192.168.15.5:String) [Test-WSMan], InvalidOperationException
    + FullyQualifiedErrorId : WsManError,Microsoft.WSMan.Management.TestWSManCommand

Same results when doing Test-NetConnection 192.168.15.5 -p 5985 (sometimes it works, sometimes it doesn't).

Server WinRM config:

Service
    RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 1500
    EnumerationTimeoutms = 240000
    MaxConnections = 300
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = false
    Auth
        Basic = true
        Kerberos = true
        Negotiate = true
        Certificate = false
        CredSSP = true
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = *
    IPv6Filter = *
    EnableCompatibilityHttpListener = false
    EnableCompatibilityHttpsListener = false
    CertificateThumbprint
    AllowRemoteAccess = true

A few observations:

1. SMB or RDP connections to the server work without any issues
2. Unifi firewall has been configured correctly
3. Windows Server firewall should have been configured correctly (otherwise I would be amazing that "sometimes" the requests work)
4. netstat -aon on the server shows it's listening correctly on the right port (5985)

Any idea what could be the case and why WinRM / Remote PowerShell is not working reliably?

Could it be Unifi having issues to reliably handle traffic over VLAN (but then why is RDP working flawlessly)?