Score:0

Linux servers stopped being able to use PowerShell remoting over WinRM using NTLM on remote Windows Servers

sm flag

So basically we have a team with Linux servers running PowerShell Core (a mix of 6 and 7) that have to execute remote Windows PowerShell commands on Windows servers. Problem is, the team never got Kerberos working for authentication and is relying on gssapi for NTLM auth to work, which AFAIK is not actually supported by MS for this scenario.

What is happening now is these Linux servers are getting MI_RESULT_FAILED errors after a while (this just started last week, things were working fine prior) and requires a restart of the WinRM service on the remote node in order for remoting to work. This works for about a day before WinRM needs to be bounced again.

Now, since NTLM isn't an auth mechanism supported by PowerShell Core on Linux (only works due to gssntlmssp which is maintained by RedHat, not Microsoft), the clear path forward here would be to either use OpenSSH for PS Remoting from Linux instead, or pivot to using Kerberos authentication instead of NTLM. However, since things were working before last week, management does not want to entertain an authentication or protocol change to resolve this issue, they want what was working to be fixed. At this point I'm spinning my wheels trying to figure out why NTLM auth broke, and my recommended fixes are off the table for now.

Source nodes are running RHEL8 or AL2, remote nodes are Windows 2016-2019, with a few legacy 2012s in the mix. Windows to Windows PS Remoting does not break here, only Linux to Windows.

Has anybody hit this situation before or something similar? Any ideas on what we might try to resolve this?

EDIT: I've corrected the error above to MI_RESULT_FAILED. ACCESS_DENIED is what the team who reported the issue said, but in our reproduction of the issue we see MI_RESULT_FAILED. WinRM trace shows NTLM auth is succeeding but still fails even when simply displaying "Hello World" via Invoke-Command.

cn flag
What is in the Windows Remote Management/Operational event log? That usually has details for the failure.
Bender the Greatest avatar
sm flag
I corrected the error above, but when the issue occurs the only log entry I get to `Windows Remote Management/Operational` is "Creating WSMan shell on server with ResourceUri: %1"
duct_tape_coder avatar
cn flag
Just to check, do you actually mean NTLM or NTLMv2? Generally NTLMv1 is disabled for security reasons.
Bender the Greatest avatar
sm flag
Sorry, they are using NTLMv2. We do have NTLMv1 disabled
Monticola Explorator avatar
sa flag
Where you able to figure this one out? I was facing the same problem, and [temporarily fixed it by setting WinRM CbtHardeningLevel to None on the server](https://github.com/jborean93/omi/issues/29)
Bender the Greatest avatar
sm flag
Nope. Magically started working on its own. We've since begun the rollout of SSH on Windows since this was MS' suggestion and gets us away from an unsupported setup.
Score:0
gt flag
Tom

I have encountered the exact same issue, our specific error was MI_RESULT_FAILED. Everything worked fine after restarting the WinRM service, but the next day it stopped working again.

It turned out to be antivirus blocking the WinRM service, when a scripted powershell request from AWX Ansible came through, because AWX used base64 encoding for the command, which triggered the 'Powershell Command Restriction - EncodedCommand' rule. That command was sent twice overnight, so until the WinRM was restarted in the morning, all communication through that PID was blocked by our AV. Hope it helps!


mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.