I work for an MSP. One of my customers has a setup that looks vaguely like this:
- The "source of truth" for their users is an on-premise Active Directory domain
- They sync their users from the on-prem AD to Azure AD
- All of their "on-premise" servers are VMs hosted in Azure
- All of their devices are Azure AD joined and are managed by Intune
- Devices are not joined to the on-premise Active Directory
- All devices, whether the user is working from home or in an office, connects back to their on-premise infrastructure, such as it is, with a VPN tunnel. They use Cisco AnyConnect to connect to a Cisco ASA VPN concentrator, also hosted in Azure
- All user devices are running Windows 10 Enterprise, 21H2.
I have an issue which is confusing me and infuriating me in equal measures. The customer still has a printer server running on Windows Server 2016. If the user needs to print, they can connect to the server by opening File Explorer, typing in the path of the printer, right clicking on the printer queue and pressing "Connect". This has never been a fast process, it could always take up to a minute to map the queue but it generally got there in an acceptable amount of time. However, I've had a lot of tickets complaining that people can't print.
I've done some investigation and have found that the issue seems to stem from amount of time that it takes to map the printer queue. When you click "Connect" to map the queue, the computer either appears to sit there and do nothing and then suddenly the queue appears 20 minutes later, or the "Connecting to printer" dialog box appears, spins its wheels for 20 minutes and then the dialog box comes up with the "downloading driver" details and maps the queue. Using the "Add-Printer" cmdlet in PowerShell also takes 20 minutes. I have used the "Measure-Command" cmdlet to time how long the Add-Printer cmdlet takes to execute and it's always 20 minutes ± 30 seconds. However, once the queue is mapped and you remove it, it maps almost immediately. Reboot the machine and it takes 20 minutes again. I've tried this on three separate devices now and it's the same on each one.
In terms of troubleshooting I've:
- Rebooted both server and client - no effect
- Checked patch levels - the server is running the September patch but the issue was first reported about 10 days after the patch was installed. The customer is usually pretty good at reporting issues like these, so I don't think it's anything to do with that. The clients are all running different patches, one on the August patch, one on September's and one on October's and it's the same on all of them.
- checked the SMBClient and SMBServer event logs on the server and the workstation and there's nothing in those relating to the devices I'm trying from
- Had our networking team check the firewall between the VPN segment and the segment the server lives in. Ports 445 and 135 are allowed through, and you can see traffic on both ports flowing from the client to the server on both of those ports
- Checked to see if the Windows firewall on the Server and Client are set to allow traffic on those ports - they both are.
- Run the "Get-TCPConnection" cmdlet on the server to verify that the two devices are communicating - they are.
- Run the "Get-SMBSession" cmdlet on the server to see if an SMB session is established. This is where it's a bit odd. The client initiates a session with the server pretty much immediately. It's authenticated with the correct user and lasts about 10-15 seconds. The session is then terminated and then, 20 minutes later, the session gets reinitiated, and the printer queue maps.
- Run Wireshark on the client device to try and get a better idea of what it's doing. When the map is first requested, there's a quick burst of traffic. You can see it authenticating with the remote server, requesting the map then you see a [RST, ACK] on port 445. It does nothing for ten minutes or so and there's another [RST, ACK] on port 135. Then, 20 minutes later, it renegotiates a connection with the server, goes through a similar sequence of authentication and requesting the printer and this time it does it successfully. I have to admit, I don't know enough about reading Wireshark traces to make much more sense of it.
I'm certain it's nothing to do with Print Nightmare, although the servers and clients have the patches which introduced the problems, we have the AdminBypass registry keys present on the clients and the RpcAuthnLevelPrivacyEnabled key on the server.
Can anyone help me figure out what it's doing for those 20 minutes and how to stop it from hanging about for so long?
