Join Log in
Score:0
linuxnoobie avatar Ubuntu

Clock skew detected on Ceph node

ml flag
linuxnoobie

I created a 3 node ceph cluster and I am getting a warning: clock skew detected on mon.ceph-node01.

All 3 nodes are in sync though (the diffences are due to time to execute and within the 5 sec Ceph threshold): #for NODE in ceph-node01 ceph-node02 ceph-node03; do ssh $NODE "hostname && date && hwclock"; done ceph-node01 Tue Nov 1 06:54:29 PM EDT 2022 2022-11-01 18:54:29.450905-04:00 ceph-node02 Tue Nov 1 06:54:30 PM EDT 2022 2022-11-01 18:54:30.435416-04:00 ceph-node03 Tue Nov 1 06:54:31 PM EDT 2022 2022-11-01 18:54:31.466657-04:00

My timedatectl shows: #timedatectl Local time: Tue 2022-11-01 18:56:33 EDT Universal time: Tue 2022-11-01 22:56:33 UTC RTC time: Tue 2022-11-01 22:56:34 Time zone: America/New_York (EDT, -0400) System clock synchronized: no NTP service: active RTC in local TZ: no

I can't get the system clock to synchronize.

I am getting this error regardless of what ntp server i choose and there are no firewalls active on the ubuntu vms: #systemctl status systemd-timesyncd.service ● systemd-timesyncd.service - Network Time Synchronization Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2022-11-01 18:42:37 EDT; 13min ago Docs: man:systemd-timesyncd.service(8) Main PID: 17846 (systemd-timesyn) Status: "Idle." Tasks: 2 (limit: 4575) Memory: 1.3M CPU: 80ms CGroup: /system.slice/systemd-timesyncd.service └─17846 /lib/systemd/systemd-timesyncd

Nov 01 18:49:05 ceph-node01 systemd-timesyncd[17846]: Timed out waiting for reply from 185.125.190.57:123 (ntp.ubuntu.com). Nov 01 18:49:16 ceph-node01 systemd-timesyncd[17846]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com). Nov 01 18:49:26 ceph-node01 systemd-timesyncd[17846]: Timed out waiting for reply from 185.125.190.56:123 (ntp.ubuntu.com).

This morning when I checked the cluster, node01 is now now skewed but node02 is. 
root@ceph-node01:/# ceph -s

cluster: id: 8536b7b0-5a10-11ed-9395-3fe791c63d1d health: HEALTH_WARN clock skew detected on mon.ceph-node02

services: mon: 2 daemons, quorum ceph-node03,ceph-node02 (age 5m) mgr: sf-ceph-node01.lmmfpp(active, since 17h), standbys: sceph-node02.mgrkgm osd: 3 osds: 3 up (since 16h), 3 in (since 18h)

data: pools: 1 pools, 1 pgs objects: 2 objects, 449 KiB usage: 19 MiB used, 3.0 TiB / 3.0 TiB avail pgs: 1 active+clean

I am now really stumped.

346
0 + 1
ntp
Christian Ehrhardt avatar
sl flag
Christian Ehrhardt
I have no solution from the content you've written, but for the sake of "trying something else" could you try to install "chrony". You do not need any of its server capabilities, but installing it will make chrony the default NTP client. And that should either work or at least give us a different POV at whatever the underlying issue is. Also chronys output on the current status is better - try `chronyc tracking` `chronyc sources` `chronyc sourcestats`
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.