It's not about the 'hosts' file – it breaks shares accessed through a different hostname than the server's "real" name. Your result is normal, as that's literally the whole purpose of the GPO.
The GPO in question is a lot like TLS SNI enforcement found in some web servers: the client states "I'm here to talk to server SRV01.EXAMPLE.COM" and if the server doesn't recognize that name as one of the vhosts it serves, it rejects the client completely. So here if the client states that it wants to talk to one of your /etc/hosts aliases, but the server isn't aware of the alias, it rejects the client.
Kerberos already has similar enforcement built-in (that's why you need to register SPNs for aliases using setspn), but the GPO makes it stricter and extends the same protection to NTLM as well.
(As already mentioned in the Docs page, this is supposed to prevent NTLM relay attacks, where a malicious server A accepts NTLM authentication then forwards the exact same NTLM packets to a real server B – the attack would be prevented because server B would see "I want to talk to server A" from the client.)
It is possible to use host aliases still, but it requires an ever-increasing amount of registry knobs. The official method is to use netdom computername REALHOST /add:THEALIAS as in this TechNet post, and I've found blog posts claiming that it is sufficient to make the aliases usable even with strict SPN validation, but it really seems that it is not quite enough – there's at least one if not two registry values that it forgets to touch.
NETDOM registers Kerberos SPNs for the alias in AD, allowing Kerberos tickets to be obtained for them, similar to manually issuing these two commands:
setspn -S HOST/THEALIAS REALHOST$
setspn -S HOST/thealias.example.com REALHOST$
This is required for Kerberos and there's no excuse to not use Kerberos in an AD environment, so whether you choose to use NETDOM or do it by hand, you should register the alias SPNs anyway.
(Technically SMB uses "cifs/" principals, but in AD they're implicitly present whenever the "host/" SPN is registered.)
Then NETDOM adds the alias in two places the server's registry, causing the server to register its additional names in AD DNS as well as announce them via NetBIOS Browser:
Path: HKLM:\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters
Name: AlternateComputerNames
Type: REG_MULTI_SZ
Data: {thealias.example.com}
Path: HKLM:\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
Name: OptionalNames
Type: REG_MULTI_SZ
Data: {THEALIAS}
They don't seem to have any effect on authentication (not even Kerberos), so I believe both of those are completely optional if you use /etc/hosts or create manual CNAME records in DNS.
One additional parameter that NETDOM doesn't create, but I've found necessary for accessing the aliases from within the server itself, is:
- Path:
HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
Name: BackConnectionHostNames
Type: REG_MULTI_SZ
Data: {thealias.example.com}
This one is only needed for "loopback" connections, so perhaps not strictly necessary in general, but at least in my case the server did need to connect to its own alias so the setting was required.
Finally, I found that the "SPN target name validation" feature has its own list of allowed names that must be manually updated – otherwise the aliases would be rejected due to strict SPN validation even with all of the above settings present:
- Path:
HKLM:\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
Name: SrvAllowedServerNames
Type: REG_MULTI_SZ
Data: {THEALIAS, thealias.example.com}
You'll want to list both the short names and FQDNs of every alias – from my experiments, listing one doesn't automatically imply the other.
With the Kerberos SPNs registered and the SrvAllowedServerNames registry value set, the aliases should at last work properly even with strict target validation.
