I recently upgraded my remote Linode, an Ubuntu 18.0.4 LTS server, to Ubuntu 22.0.4 LTS.
Ever since then, Lassie keeps rebooting my server, roughly, every 15 minutes. I have looked into both /var/log/syslog and `/var/log/kern.log - here are the contents:
[[Note]]: I have only included parts of the log file that seemed significant (to avoid having to post entire log file contents here).
syslog
Dec 7 00:00:01 ceberus CRON[3612]: (root) CMD (/usr/bin/certbot renew --quiet)
Dec 7 00:00:01 ceberus CRON[3613]: (erpnext) CMD (cd /opt/bench/erpnext && /usr/local/bin/bench --verbose --site all backup >> /opt/bench/erpnext/logs/backup.log 2>&1 # bench auto backups set for every 6 hours)
Dec 7 00:00:01 ceberus CRON[3614]: (erpnext) CMD (cd /home/erpnext/frappe-bench && /usr/local/bin/bench --verbose --site all backup >> /home/erpnext/frappe-bench/logs/backup.log 2>&1 # bench auto backups set for every 6 hours)
Dec 7 00:00:01 ceberus CRON[3615]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
Dec 7 00:00:01 ceberus systemd[1]: Starting Daily dpkg database backup service...
Dec 7 00:00:01 ceberus systemd[1]: Starting Rotate log files...
Dec 7 00:00:01 ceberus CRON[3621]: (erpnext) CMD (cd /opt/bench/erpnext && /opt/bench/env/bin/bench --verbose --site all backup >> /opt/bench/erpnext/logs/backup.log 2>&1 # bench auto backups set for every 6 hours)
Dec 7 00:00:02 ceberus kernel: [ 453.683560] [UFW BLOCK] IN=eth0 OUT= MAC=f2:3c:91:4f:81:af:84:78:ac:1c:ec:41:86:dd SRC=2a06:4880:4000:0000:0000:0000:0000:0061 DST=2600:3c00:0000:0000:f03c:91ff:fe4f:81af LEN=64 TC=40 HOPLIMIT=243 FLOWLBL=27227 PROTO=TCP SPT=43519 DPT=33012 WINDOW=14600 RES=0x00 SYN URGP=0
Dec 7 00:00:02 ceberus systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Dec 7 00:00:02 ceberus systemd[1]: Finished Daily dpkg database backup service.
Dec 7 00:00:02 ceberus postfix/pickup[2505]: 1740DDC485: uid=1002 from=<erpnext>
Dec 7 00:00:02 ceberus postfix/cleanup[3645]: 1740DDC485: message-id=<20221207000002.1740DDC485@ceberus.members.linode.com>
Dec 7 00:00:02 ceberus postfix/qmgr[2506]: 1740DDC485: from=<erpnext@ceberus.members.linode.com>, size=761, nrcpt=1 (queue active)
Dec 7 00:00:02 ceberus postfix/pickup[2505]: 23F16DC3BA: uid=1002 from=<erpnext>
Dec 7 00:00:02 ceberus postfix/qmgr[2506]: 23F16DC3BA: from=<erpnext@ceberus.members.linode.com>, size=785, nrcpt=1 (queue active)
Dec 7 00:00:02 ceberus postfix/pickup[2505]: 29FD6DC3BC: uid=1002 from=<erpnext>
Dec 7 00:00:02 ceberus postfix/cleanup[3645]: 29FD6DC3BC: message-id=<20221207000002.29FD6DC3BC@ceberus.members.linode.com>
Dec 7 00:00:02 ceberus postfix/qmgr[2506]: 29FD6DC3BC: from=<erpnext@ceberus.members.linode.com>, size=765, nrcpt=1 (queue active)
Dec 7 00:00:02 ceberus systemd[1]: Started snap.certbot.certbot.2d0b6daa-1d1a-4726-ae2a-083c23951938.scope.
Dec 7 00:00:02 ceberus postfix/local[3649]: 1740DDC485: to=<erpnext@ceberus.members.linode.com>, orig_to=<erpnext>, relay=local, delay=0.39, delays=0.26/0.12/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Dec 7 00:00:02 ceberus postfix/qmgr[2506]: 1740DDC485: removed
Dec 7 00:00:02 ceberus postfix/local[3649]: 23F16DC3BA: to=<erpnext@ceberus.members.linode.com>, orig_to=<erpnext>, relay=local, delay=0.4, delays=0.27/0.12/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Dec 7 00:00:02 ceberus postfix/qmgr[2506]: 23F16DC3BA: removed
Dec 7 00:00:02 ceberus postfix/local[3649]: 29FD6DC3BC: to=<erpnext@ceberus.members.linode.com>, orig_to=<erpnext>, relay=local, delay=0.44, delays=0.29/0.12/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Dec 7 00:00:02 ceberus postfix/qmgr[2506]: 29FD6DC3BC: removed
Dec 7 00:00:02 ceberus systemd[1]: Stopping Make remote CUPS printers available locally...
Dec 7 00:00:02 ceberus systemd[1]: cups-browsed.service: Deactivated successfully.
Dec 7 00:00:02 ceberus systemd[1]: Stopped Make remote CUPS printers available locally.
Dec 7 00:00:02 ceberus systemd[1]: Stopping CUPS Scheduler...
Dec 7 00:00:02 ceberus systemd[1]: docker-957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9.scope: Deactivated successfully.
Dec 7 00:00:02 ceberus systemd[1]: docker-957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9.scope: Consumed 9.037s CPU time.
Dec 7 00:00:02 ceberus dockerd[1235]: time="2022-12-07T00:00:02.692286512Z" level=info msg="ignoring event" container=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 7 00:00:02 ceberus containerd[1055]: time="2022-12-07T00:00:02.696700524Z" level=info msg="shim disconnected" id=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9
02e24669d20133da47cbefbef1ac655a8fc46fd95d9 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 7 00:00:02 ceberus containerd[1055]: time="2022-12-07T00:00:02.696700524Z" level=info msg="shim disconnected" id=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9
Dec 7 00:00:02 ceberus containerd[1055]: time="2022-12-07T00:00:02.711375120Z" level=warning msg="cleaning up after shim disconnected" id=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9 namespace=moby
Dec 7 00:00:02 ceberus containerd[1055]: time="2022-12-07T00:00:02.712001642Z" level=info msg="cleaning up dead shim"
Dec 7 00:00:02 ceberus containerd[1055]: time="2022-12-07T00:00:02.757962688Z" level=warning msg="cleanup warnings time=\"2022-12-07T00:00:02Z\" level=info msg=\"starting signal loop\" namespace=moby pid=3675 runtime=io.containerd.runc.v2\n"
Dec 7 00:00:02 ceberus systemd[1]: mnt-startup_hub_volume-var-lib-docker-overlay2-597f724566a93c983ea6857579b541d3ca102c4c0191bd030404e42d636b9d5a-merged.mount: Deactivated successfully.
Dec 7 00:12:34 ceberus systemd[1]: run-docker-runtime\x2drunc-moby-957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9-runc.PGShN2.mount: Deactivated successfully.
Dec 7 00:12:42 ceberus systemd[1]: docker-957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9.scope: Deactivated successfully.
Dec 7 00:12:42 ceberus systemd[1]: docker-957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9.scope: Consumed 8.193s CPU time.
Dec 7 00:12:42 ceberus dockerd[1235]: time="2022-12-07T00:12:42.936044176Z" level=info msg="ignoring event" container=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 7 00:12:42 ceberus containerd[1055]: time="2022-12-07T00:12:42.937086374Z" level=info msg="shim disconnected" id=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9
Dec 7 00:12:42 ceberus containerd[1055]: time="2022-12-07T00:12:42.937551034Z" level=warning msg="cleaning up after shim disconnected" id=957e78ab60b5025c0233702e24669d20133da47cbefbef1ac655a8fc46fd95d9 namespace=moby
Dec 7 00:12:42 ceberus containerd[1055]: time="2022-12-07T00:12:42.937730763Z" level=info msg="cleaning up dead shim"
Dec 7 00:12:42 ceberus containerd[1055]: time="2022-12-07T00:12:42.948888416Z" level=warning msg="cleanup warnings time=\"2022-12-07T00:12:42Z\" level=info msg=\"starting signal loop\" namespace=moby pid=5266 runtime=io.containerd.runc.v2\n"
Dec 7 00:12:42 ceberus containerd[1055]: time="2022-12-07T00:12:42.948888416Z" level=warning msg="cleanup warnings time=\"2022-12-07T00:12:42Z\" level=info msg=\"starting signal loop\" namespace=moby pid=5266 runtime=io.containerd.runc.v2\n"
Dec 7 00:12:43 ceberus systemd[1]: mnt-startup_hub_volume-var-lib-docker-overlay2-597f724566a93c983ea6857579b541d3ca102c4c0191bd030404e42d636b9d5a-merged.mount: Deactivated successfully.
Dec 7 00:12:50 ceberus NetworkManager[579]: <info> [1670371970.4317] manager: sleep: sleep requested (sleeping: no enabled: yes)
Dec 7 00:12:50 ceberus gnome-shell[1226]: Screen lock is locked down, not locking
Dec 7 00:12:50 ceberus ModemManager[713]: <info> [sleep-monitor] system is about to suspend
Dec 7 00:12:50 ceberus systemd[1]: Reached target Sleep.
Dec 7 00:12:50 ceberus systemd[1]: Starting Record successful boot for GRUB...
Dec 7 00:12:50 ceberus systemd[1]: Starting System Suspend...
Dec 7 00:12:50 ceberus ModemManager[713]: <info> [sleep-monitor] system is about to suspend
Dec 7 00:12:50 ceberus systemd[1]: Reached target Sleep.
Dec 7 00:12:50 ceberus systemd[1]: Starting Record successful boot for GRUB...
Dec 7 00:12:50 ceberus systemd[1]: Starting System Suspend...
Dec 7 00:12:50 ceberus systemd[1]: grub-common.service: Deactivated successfully.
Dec 7 00:12:50 ceberus systemd[1]: Finished Record successful boot for GRUB.
Dec 7 00:12:50 ceberus systemd[1]: Starting GRUB failed boot detection...
Dec 7 00:12:50 ceberus systemd[1]: grub-initrd-fallback.service: Deactivated successfully.
Dec 7 00:12:50 ceberus systemd[1]: Finished GRUB failed boot detection.
Dec 7 00:12:50 ceberus systemd-sleep[5283]: Entering sleep state 'suspend'...
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Dec 7 00:13:31 ceberus systemd-modules-load[371]: Failed to find module 'lp'
Dec 7 00:13:31 ceberus systemd-modules-load[371]: Failed to find module 'ppdev'
Dec 7 00:13:31 ceberus systemd-resolved[439]: Using system hostname 'CEBERUS'.
Dec 7 00:13:31 ceberus systemd[1]: Started Network Name Resolution.
Dec 7 00:13:31 ceberus systemd-udevd[446]: Using default interface naming scheme 'v249'.
Dec 7 00:13:31 ceberus systemd-udevd[426]: sdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb' failed with exit code 1.
Dec 7 00:13:31 ceberus systemd-udevd[445]: sda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda' failed with exit code 1.
Dec 7 00:13:31 ceberus systemd[1]: Found device /dev/ttyS0.
Dec 7 00:13:31 ceberus systemd-udevd[429]: sdc: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdc' failed with exit code 1.
Dec 7 00:13:31 ceberus systemd[1]: Found device Virtio network device.
Dec 7 00:13:31 ceberus systemd[1]: Reached target System Initialization.
Dec 7 00:13:31 ceberus systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (file watch) being skipped.
Dec 7 00:13:31 ceberus systemd[1]: Started CUPS Scheduler.
Dec 7 00:13:31 ceberus systemd[1]: Started resolvconf-pull-resolved.path.
Dec 7 00:13:31 ceberus systemd[1]: whoopsie.path: Refusing to start, unit whoopsie.service to trigger not loaded.
Dec 7 00:13:31 ceberus systemd[1]: Failed to start Start whoopsie on modification of the /var/crash directory.
Dec 7 00:13:31 ceberus systemd[1]: Started Trigger anacron every hour.
Dec 7 00:13:31 ceberus systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (timer based) being skipped.
kern.log
Dec 7 00:11:07 ceberus kernel: [ 1118.799235] [UFW BLOCK] IN=eth0 OUT= MAC=REDACTED SRC=92.63.196.38 DST=45.33.121.232 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=41428 PROTO=TCP SPT=53496 DPT=4749 WINDOW=1024 RES=0x00 SYN URGP=0
Dec 7 00:11:28 ceberus kernel: [ 1140.500279] [UFW BLOCK] IN=eth0 OUT= MAC=REDACTED SRC=176.111.174.81 DST=45.33.121.232 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=48644 PROTO=TCP SPT=46148 DPT=33388 WINDOW=1024 RES=0x00 SYN URGP=0
Dec 7 00:12:06 ceberus kernel: [ 1177.695725] [UFW BLOCK] IN=eth0 OUT= MAC=REDACTED SRC=222.186.170.196 DST=45.33.121.232 LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=25486 PROTO=TCP SPT=33446 DPT=4300 WINDOW=63540 RES=0x00 SYN URGP=0
Dec 7 00:12:24 ceberus kernel: [ 1196.427023] [UFW BLOCK] IN=eth0 OUT= MAC=REDACTED SRC=79.124.62.130 DST=45.33.121.232 LEN=40 TOS=0x08 PREC=0x40 TTL=234 ID=23062 PROTO=TCP SPT=47912 DPT=4000 WINDOW=1024 RES=0x00 SYN URGP=0
Dec 7 00:12:25 ceberus kernel: [ 1197.072018] [UFW BLOCK] IN=eth0 OUT= MAC=REDACTED SRC=212.70.149.46 DST=45.33.121.232 LEN=40 TOS=0x08 PREC=0x40 TTL=234 ID=54321 PROTO=TCP SPT=59239 DPT=8443 WINDOW=65535 RES=0x00 SYN URGP=0
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Dec 7 00:13:31 ceberus kernel: [ 0.000000] Linux version 6.0.2-x86_64-linode157 (maker@build.linode.com) (gcc (Debian 8.3.0-6) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #1 SMP PREEMPT_DYNAMIC Mon Oct 17 13:01:41 EDT 2022
Dec 7 00:13:31 ceberus kernel: [ 0.000000] Command line: root=/dev/sda console=tty1 console=ttyS0 ro devtmpfs.mount=1
Dec 7 00:13:31 ceberus kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Dec 7 00:13:31 ceberus kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Dec 7 00:13:31 ceberus kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Dec 7 00:13:31 ceberus kernel: [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Dec 7 00:13:31 ceberus kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format.
Dec 7 00:13:31 ceberus kernel: [ 0.000000] signal: max sigframe size: 1776
Dec 7 00:13:31 ceberus kernel: [ 0.000000] BIOS-provided physical RAM map:
Dec 7 00:13:31 ceberus kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Dec 7 00:13:31 ceberus kernel: [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Dec 7 00:13:31 ceberus kernel: [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Dec 7 00:13:31 ceberus kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffd6fff] usable
Dec 7 00:13:31 ceberus systemd[1]: Reached target System Initialization.
Dec 7 00:13:31 ceberus systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (file watch) being skipped.
Dec 7 00:13:31 ceberus systemd[1]: Started CUPS Scheduler.
Dec 7 00:13:31 ceberus systemd[1]: Started resolvconf-pull-resolved.path.
Dec 7 00:13:31 ceberus systemd[1]: whoopsie.path: Refusing to start, unit whoopsie.service to trigger not loaded.
Dec 7 00:13:31 ceberus systemd[1]: Failed to start Start whoopsie on modification of the /var/crash directory.
Dec 7 00:13:31 ceberus systemd[1]: Started Trigger anacron every hour.
Dec 7 00:13:31 ceberus systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (timer based) being skipped.
...
I don't know enough about the contents of syslog and kern.log to be able to discern if there is any information in here, that explains why the server is being restarted - and equally (if not more) importantly, how to FIX this issue.
