My Ubuntu box has begun to freeze intermittently, and I want to understand how I can start to diagnose and troubleshoot this?
- The machine is acting as a server, sitting in my garage, hosting a small number of web applications through Docker.
- Every once in a while, the machine becomes totally unresponsive. It does not answer web requests, and I cannot even access it by SSH. After ten minutes or so, everything is fine again.
- When I then log into the machine via SSH, I can see by the
top command that something is bogging it down hard: the normal load is <0.5 but sometimes, something is pushing it over 170! No wonder it looks to be frozen!
Update:
dmesg -T | grep ill shows that one of the Docker containers was recently killed by the system, and the same happened yesterday. That would coincide with the time when the machine became responsive again.
cat /var/log/syslog did not have entries around this tiestamp.
Evidently there is a problem with a process named mattermost. This is where I must investigate further.
[Do Mär 2 12:18:39 2023] [ 3858] 1000 3858 113617 243 106496 5 0 gsd-rfkill
[Do Mär 2 12:18:39 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/4fef02a344354214b6acb6f36102bc59e9dc50557d5c2632d70ab3daac100a5f,task=bundle,pid=1538755,uid=998
[Do Mär 2 12:18:39 2023] Out of memory: Killed process 1538755 (bundle) total-vm:1578372kB, anon-rss:904568kB, file-rss:0kB, shmem-rss:1956kB, UID:998 pgtables:2900kB oom_score_adj:0
[Do Mär 2 12:19:23 2023] nxnode.bin invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Do Mär 2 12:19:23 2023] oom_kill_process.cold+0xb/0x10
[Do Mär 2 12:19:23 2023] [ 3858] 1000 3858 113617 243 106496 5 0 gsd-rfkill
[Do Mär 2 12:19:23 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/83ee973eca02114fae81d46c67fd021c6fad40eb124609eff8f6067ff22335dc,task=mattermost,pid=3800325,uid=2000
[Do Mär 2 12:19:23 2023] Out of memory: Killed process 3800325 (mattermost) total-vm:25852356kB, anon-rss:4436936kB, file-rss:0kB, shmem-rss:0kB, UID:2000 pgtables:9268kB oom_score_adj:0
[Fr Mär 3 10:31:08 2023] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Fr Mär 3 10:31:08 2023] oom_kill_process.cold+0xb/0x10
[Fr Mär 3 10:31:08 2023] [ 3858] 1000 3858 113617 245 106496 5 0 gsd-rfkill
[Fr Mär 3 10:31:08 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/6ceafe411e36e8c1ceb396ae0fe9e0772399b4d981f0c3b9f108c4b637300d6f,task=bundle,pid=2216864,uid=998
[Fr Mär 3 10:31:08 2023] Out of memory: Killed process 2216864 (bundle) total-vm:1627016kB, anon-rss:922008kB, file-rss:0kB, shmem-rss:1956kB, UID:998 pgtables:2932kB oom_score_adj:0
[Fr Mär 3 10:31:46 2023] nxnode.bin invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Fr Mär 3 10:31:46 2023] oom_kill_process.cold+0xb/0x10
[Fr Mär 3 10:31:46 2023] [ 3858] 1000 3858 113617 245 106496 5 0 gsd-rfkill
[Fr Mär 3 10:31:46 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/66ecc2904f86807abf8e5a3e5fb9310028a03c19d09f6b8973eb2c6801c943f7,task=mattermost,pid=2425866,uid=2000
[Fr Mär 3 10:31:46 2023] Out of memory: Killed process 2425866 (mattermost) total-vm:101008668kB, anon-rss:4685940kB, file-rss:0kB, shmem-rss:0kB, UID:2000 pgtables:10188kB oom_score_adj:0
