Having a Windows Server 2016 (hosted at Amazon AWS EC2) that is running successfully for years.
Since some month I'm experiencing that the Scheduled Tasks (e.g. a task that is configured to run daily) are not being executed. No error message in the task history. No entry in the Event Log. Nothing.
The tasks simply do not run, and in the "Next Run Time" column is a date in the future, e.g. tomorrow. When looking again tomorrow, the same picture: not ran any tasks, "Next Run Time" points to the future, again.
Here is another example of what the odds in my system time look like:
Above, a screenshot of Windows Update shows that an update of SQL Server that I've installed at 5th of February 2022 actually is erroneously being shown as installed at 5th of March 2022. One month in the future.
I've just called this in the command line:
C:\>w32tm /query /status
Which printed this result:
Leap Indicator: 0(no warning)
Stratum: 4 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0096024s
Root Dispersion: 0.0493815s
ReferenceId: 0x28779426 (source IP: 40.119.148.38)
Last Successful Sync Time: 08.02.2022 08:41:13
Source: time.windows.com,0x8
Poll Interval: 10 (1024s)
Nothing suspicious looking to me.
I've also checked for viruses without any positives. I've also run the "MRT" tool without any positives.
The last time this happend, maybe a month ago, I've rebooted my system and I guess this (temporarily) solved the issue. Just to appear now again.
Having multiple other EC2 AWS Windows 2016 servers, and dozens of other Windows servers to administer for years, I never came across such a weird behavior.
My question
Does anyone have a clue what might go on on this server and how to resolve this?
Update 1
It seems that this first occurred after an automatic daylight-saving time switch (the server and time zone is in Germany).
Some of the tasks had this message as the "Last Run Result":
The operator or administrator has refused the request. (0x800710E0)
In German:
Die Anforderung wurde vom Operator oder Administrator zurückgewiesen. (0x800710E0)
All suggestions I found about this error (like e.g. this one) was to edit and store tasks again. I'll try this now.
Update 2
The scheduled tasks again did not run and I found multiple entries like the following in the Event Log:
The system time has changed to 2022-03-20T01:25:40.348000000Z from 2022-02-16T14:47:49.130142400Z.
Change Reason: An application or system component changed the time.
So it was changed to one month in the future.
And later, it was changed back to the correct date again:
The system time has changed to 2022-02-16T12:04:30.541000000Z from 2022-03-20T01:37:24.558311600Z.
Change Reason: An application or system component changed the time.
So this changed the date one month into the future and back again. It seems this is enough to confuse my tasks enough to not run ever again.
I still don't know the reason.
One suggestion I found regarding these Event Log entries was to re-register the time service:
net stop w32time
w32tm /unregister
w32tm /register
net start w32time
Since the server runs on EC2 at Amazon, AWS, I'll not try this command:
w32tm /config /manualpeerlist:169.254.169.123 /syncfromflags:manual /update
And also this command:
reg add "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /d 1 /t REG_DWORD /f
Then, to make the Scheduled Tasks run again, I'll open each one and save it again.
Update 3
We've now experienced time changes again, following is what the audit log entries look like.
First, the change was 133 days into the future:
The selected entry at "17.03.22 06:47:44" has these details:
A new process has been created.
Creator Subject:
Security ID: SYSTEM
Account Name: EC2AMAZ-K5BI954$
Account Domain: WORKGROUP
Logon ID: 0x3E7
Target Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0
Process Information:
New Process ID: 0x3478
New Process Name: C:\Windows\System32\conhost.exe
Token Elevation Type: %%1936
Mandatory Label: Mandatory Label\System Mandatory Level
Creator Process ID: 0x44bc
Creator Process Name: C:\Windows\System32\wbem\WMIC.exe
Process Command Line: ??\C:\Windows\system32\conhost.exe 0xffffffff -ForceV1
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.
And the first entry at the new (wrong) date in the future "28.07.22 13:52:50" was:
A new process has been created.
Creator Subject:
Security ID: SYSTEM
Account Name: EC2AMAZ-K5BI954$
Account Domain: WORKGROUP
Logon ID: 0x3E7
Target Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0
Process Information:
New Process ID: 0x1f64
New Process Name: C:\Program Files (x86)\Google\Update\GoogleUpdate.exe
Token Elevation Type: %%1936
Mandatory Label: Mandatory Label\System Mandatory Level
Creator Process ID: 0x4a8
Creator Process Name: C:\Windows\System32\svchost.exe
Process Command Line: "C:\Program Files (x86)\Google\Update\GoogleUpdate.exe" /ua /installsource scheduler
I've googled for the "??\C:\Windows\system32\conhost.exe 0xffffffff -ForceV1" and found several articles that claim that this might be malicious:
The system time was later automatically set back by the system:
In fact it was first changed back too much to 16.03.2022 and right after that to the correct time 17.03.2022 again.
The details of the entry looks like:
The system time was changed.
Subject:
Security ID: LOCAL SERVICE
Account Name: LOCAL SERVICE
Account Domain: NT AUTHORITY
Logon ID: 0x3E5
Process Information:
Process ID: 0x4a0
Name: C:\Windows\System32\svchost.exe
Previous Time: 2022-07-28T11:59:14.787568600Z
New Time: 2022-03-16T15:02:28.443000000Z
This event is generated when the system time is changed. It is normal for the Windows Time Service, which runs with System privilege, to change the system time on a regular basis. Other system time changes may be indicative of attempts to tamper with the computer.
So now, even after I've activated audit, I'm still completely confused about what is happening on my system.
Update 4
We finally gave up and set up another AWS EC2 machine, this time a Windows Server 2022.
I migrated everything manually from the old server to the new server (files, databases, IIS websites, etc.).
It is running since approx. 2 weeks now without any issues.
While this is not a technical solution, it is at least a working one.
