System clock seems off by -2m11.785152258s, which can prevent network connectivity

I have a server that was syncing a bsc full node, and all of a sudden my system clock went off by 2 minutes. Could it be a hardware problem?

OS: Ubuntu 22.04.2 LTS x86_64


❯ timedatectl status
               Local time: Sun 2023-04-09 16:29:58 UTC
           Universal time: Sun 2023-04-09 16:29:58 UTC
                 RTC time: Sun 2023-04-09 16:29:58
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
❯ sudo service systemd-timedated status                                                                                                                                                                                                                                                                       ─╯
● systemd-timedated.service - Time & Date Service
     Loaded: loaded (/lib/systemd/system/systemd-timedated.service; static)
     Active: active (running) since Sun 2023-04-09 17:32:05 UTC; 19s ago
       Docs: man:systemd-timedated.service(8)
             man:localtime(5)
             man:org.freedesktop.timedate1(5)
   Main PID: 5563 (systemd-timedat)
      Tasks: 1 (limit: 77088)
     Memory: 1.1M
        CPU: 90ms
     CGroup: /system.slice/systemd-timedated.service
             └─5563 /lib/systemd/systemd-timedated

Apr 09 17:32:05 bscscan systemd[1]: Starting Time & Date Service...
Apr 09 17:32:05 bscscan systemd[1]: Started Time & Date Service.

I have the system clock synchronization enabled and ntp service active. And still the clock shows the incorrect time. How do I debug this issue?

What I tried is the following

1. sudo ntpdate -s pool.ntp.org
2. sudo service ntp restart
3. reboot

❯ sudo ntpdate ntp.ubuntu.com
 9 Apr 16:52:50 ntpdate[3389]: step time server 185.125.190.56 offset +131.774604 sec
❯ date
Sun Apr  9 04:50:42 PM UTC 2023

UPDATE the clock difference is slowly increasing if you look at the offset

❯ sudo ntpdate ntp.ubuntu.com
 9 Apr 16:52:50 ntpdate[3389]: step time server 185.125.190.56 offset +131.774604 sec
❯ date
Sun Apr  9 04:50:42 PM UTC 2023
❯ sudo ntpdate -q ntp.ubuntu.com
server 185.125.190.56, stratum 2, offset +131.790065, delay 0.07298
server 185.125.190.57, stratum 2, offset +131.790275, delay 0.07809
server 185.125.190.58, stratum 2, offset +131.788416, delay 0.07237
server 91.189.91.157, stratum 2, offset +131.792651, delay 0.14906
server 91.189.94.4, stratum 2, offset +131.785681, delay 0.09180
 9 Apr 16:52:06 ntpdate[3509]: step time server 185.125.190.58 offset +131.788416 sec
❯ sudo ntpdate md.pool.ntp.org
 9 Apr 16:55:33 ntpdate[3592]: step time server 185.177.151.75 offset +131.790053 sec
❯ date
Sun Apr  9 04:53:23 PM UTC 2023
❯ sudo ntpdate md.pool.ntp.org
 9 Apr 16:57:25 ntpdate[3670]: step time server 185.177.151.86 offset +131.791959 sec

I managed to fix it by runnging:

sudo systemctl stop ntp
sudo ntpd -gq
sudo systemctl start ntp