Join Log in
Score:0
Witty Tut avatar Ubuntu

Ubuntu 22.04 server crashes randomly

co flag
Witty Tut

Ubuntu 22.04 server crashes randomly. I'm unsure how to get to the fault of the issue and am willing to provide whatever logs are needed.

I don't see anything of help in /var/crash or /var/log.

How do I fix this issue?

Edit: I believe the following is the part in the log where it crashed this morning around 10am. Nmath is this all the requested information?

Dec 16 09:59:41 servertut kernel: [  529.068579] [  12305]  1000 12305     9572      669   126976      865             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068581] [  12328]  1000 12328     9090     1291   114688      147             0 panel-6-systray
Dec 16 09:59:41 servertut kernel: [  529.068584] [  12329]  1000 12329   533353   108792  1122304    16905             0 panel-9-power-m
Dec 16 09:59:41 servertut kernel: [  529.068587] [  12354]  1000 12354     9556     1521   126976        8             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068589] [  12375]  1000 12375     9559      447   118784     1089             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068592] [  12390]  1000 12390     9562     1241   114688      294             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068595] [  12408]  1000 12408     9553      306   114688     1220             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068597] [  12428]  1000 12428     9552     1045   114688      489             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068601] [  12506]  1000 12506     9088      988   110592      448             0 panel-6-systray
Dec 16 09:59:41 servertut kernel: [  529.068604] [  12515]  1000 12515   533353   484990  4231168    30224             0 panel-9-power-m
Dec 16 09:59:41 servertut kernel: [  529.068607] [  12547]  1000 12547     9553      662   118784      871             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068609] [  12611]  1000 12611     9089     1227   114688      213             0 panel-6-systray
Dec 16 09:59:41 servertut kernel: [  529.068612] [  12623]  1000 12623   533353    38831   479232     6335             0 panel-9-power-m
Dec 16 09:59:41 servertut kernel: [  529.068615] [  12671]  1000 12671     9553     1040   114688      488             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068618] [  12756]  1000 12756     9552     1404   122880      129             0 panel-14-action
Dec 16 09:59:41 servertut kernel: [  529.068623] [  14076]     0 14076      389        1    32768        0             0 default.script
Dec 16 09:59:41 servertut kernel: [  529.068626] [  14078]     0 14078      398        1    36864        0             0 default.script
Dec 16 09:59:41 servertut kernel: [  529.068628] [  14080]     0 14080      240        1    20480        0             0 default.script
Dec 16 09:59:41 servertut kernel: [  529.068631] [  14088]     0 14088      240        1    20480        0             0 default.script
Dec 16 09:59:41 servertut kernel: [  529.068633] [  14091]     0 14091      398        1    36864        0             0 default.script
Dec 16 09:59:41 servertut kernel: [  529.068636] [  14092]     0 14092      240        1    20480        0             0 default.script
Dec 16 09:59:41 servertut kernel: [  529.068639] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-acca4b033fc8d6071a2faa8b89867805431fcc6f704a36f8984266576c42c0ed.scope,mems_allowed=0-1,global_oom,task_memcg=/system.slice/docker-f2c4f988e1bb2b628762755c114121b5079f66aaf449f374e56835399a5a1ec0.scope,task=panel-9-power-m,pid=10297,uid=1000
Dec 16 09:59:41 servertut kernel: [  529.068667] Out of memory: Killed process 10297 (panel-9-power-m) total-vm:4230568kB, anon-rss:2045468kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:6028kB oom_score_adj:0
Dec 16 10:00:03 servertut containerd[1927]: time="2022-12-16T09:59:44.048126231Z" level=error msg="publish OOM event" error="context deadline exceeded"
Dec 16 10:00:03 servertut containerd[1927]: time="2022-12-16T09:59:45.696930028Z" level=error msg="publish OOM event" error="context deadline exceeded"
Dec 16 10:00:03 servertut containerd[1927]: time="2022-12-16T09:59:47.121047563Z" level=error msg="forward event" error="context deadline exceeded"
Dec 16 10:00:09 servertut kernel: [  550.101957] ThreadPoolForeg invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 16 10:00:09 servertut kernel: [  550.101972] CPU: 6 PID: 13879 Comm: ThreadPoolForeg Kdump: loaded Not tainted 5.15.0-56-generic #62-Ubuntu
Dec 16 10:00:09 servertut kernel: [  550.101976] Hardware name: Dell Inc. Precision T5610/0WN7Y6, BIOS A19 09/11/2019
Dec 16 10:00:09 servertut kernel: [  550.101979] Call Trace:
Dec 16 10:00:09 servertut kernel: [  550.101983]  <TASK>
Dec 16 10:00:09 servertut kernel: [  550.101988]  show_stack+0x52/0x5c
Dec 16 10:00:09 servertut kernel: [  550.101997]  dump_stack_lvl+0x4a/0x63
Dec 16 10:00:09 servertut kernel: [  550.102006]  dump_stack+0x10/0x16
Dec 16 10:00:09 servertut kernel: [  550.102010]  dump_header+0x53/0x228
Dec 16 10:00:09 servertut kernel: [  550.102018]  oom_kill_process.cold+0xb/0x10
Dec 16 10:00:09 servertut kernel: [  550.102021]  out_of_memory+0x106/0x2e0
Dec 16 10:00:09 servertut kernel: [  550.102030]  __alloc_pages_slowpath.constprop.0+0x97e/0xa40
Dec 16 10:00:09 servertut kernel: [  550.102040]  __alloc_pages+0x311/0x330
Dec 16 10:00:09 servertut kernel: [  550.102044]  alloc_pages+0x9e/0x1e0
Dec 16 10:00:09 servertut kernel: [  550.102048]  __page_cache_alloc+0x7e/0x90
Dec 16 10:00:09 servertut kernel: [  550.102052]  pagecache_get_page+0x152/0x590
Dec 16 10:00:09 servertut kernel: [  550.102061]  ? page_cache_ra_unbounded+0x166/0x210
Dec 16 10:00:09 servertut kernel: [  550.102066]  filemap_fault+0x488/0xab0
Dec 16 10:00:09 servertut kernel: [  550.102069]  ? filemap_map_pages+0x309/0x400
Dec 16 10:00:09 servertut kernel: [  550.102075]  __do_fault+0x3c/0x120
Dec 16 10:00:09 servertut kernel: [  550.102082]  do_read_fault+0xeb/0x160
Dec 16 10:00:09 servertut kernel: [  550.102085]  do_fault+0xa0/0x2e0
Dec 16 10:00:09 servertut kernel: [  550.102089]  handle_pte_fault+0x1cd/0x240
Dec 16 10:00:09 servertut kernel: [  550.102096]  __handle_mm_fault+0x405/0x6f0
Dec 16 10:00:09 servertut kernel: [  550.102106]  handle_mm_fault+0xd8/0x2c0
Dec 16 10:00:09 servertut kernel: [  550.102110]  do_user_addr_fault+0x1c9/0x670
Dec 16 10:00:09 servertut kernel: [  550.102117]  exc_page_fault+0x77/0x170
Dec 16 10:00:09 servertut kernel: [  550.102125]  asm_exc_page_fault+0x27/0x30
Dec 16 10:00:09 servertut kernel: [  550.102132] RIP: 0033:0x5558f157225d
Dec 16 10:00:09 servertut kernel: [  550.102141] Code: Unable to access opcode bytes at RIP 0x5558f1572233.```
447
0 + 5
Hannu avatar
ca flag
Hannu
+ what kind of server is it; web, nas, ... do you have load on it, does it crash on top of that, ....
0
Reply
Witty Tut avatar
co flag
Witty Tut
Thank you for the feedback. At first I thought it was due to usb over current due to seeing that information in cockpit. So I followed a guide to disable the over current check [link](https://askubuntu.com/questions/908165/usb-2-1-over-current-message). I have not been able to reproduce the issue myself. It seems to completely crash (all lights go off) then boot back up. I've ran the Dell Pre post check and it all came back normal. Here is the syslog from /var/log https://drive.google.com/file/d/1rsbUkv8DLjI5t7QRL8oi8jzbS2QVFLuD/view?usp=share_link
0
Reply
Witty Tut avatar
co flag
Witty Tut
The server is running 20 or so containers in docker with a usb hub hooked up to it and 15 phones tethering through the hub. That is all I use this for.
0
Reply
waltinator avatar
it flag
waltinator
Use `sudo journalctl -b -1 -e` to see the logs just before the previous reboot. `sudo journalctl --list-boots` to see other choices for the "`-b`" switch. Read `man journalctl`.
0
Reply
Witty Tut avatar
co flag
Witty Tut
@Nmath I believe I have done what is requested. Please advise if anything else is needed.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.