This may be due to a feature in Windows Server 2016+. When time sync is temporarily unavailable, use the time stamp from TLS packets. Unfortunately, sometimes the timestamp has random data inserted by other applications for other purposes, and it should not be relied upon.
Secure Time Seeding
This feature determines the approximate current time from outgoing SSL connections. This time value is used to monitor the local system clock and correct any gross errors. In deployments with a reliable time source(s) and well monitored machines that include monitoring for time offsets, you may choose to not use the Secure Time Seeding feature and rely on your existing infrastructure instead.
You can disable the feature with these steps:
Set the UtilizeSSLTimeData registry value to 0:
reg add HKLM\SYSTEM\CurrentControlSet\Services\w32time\Config /v UtilizeSslTimeData /t REG_DWORD /d 0 /f
If you are unable to reboot the machine immediately due to some reason, you can notify W32time service about the configuration update. This stops time monitoring and enforcement based on time data collected from SSL connections.
W32tm.exe /config /update
https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-server-2016-improvements
https://learn.microsoft.com/en-us/archive/blogs/w32time/secure-time-seeding-improving-time-keeping-in-windows
https://learn.microsoft.com/en-us/troubleshoot/windows-server/identity/turn-on-debug-logging-in-windows-time-service