Score:0

System failed, please help me interpret last console messages (screen capture)

us flag

This server runs several processes of satellite imagery, it has 256GB of RAM, 12TB disk, 64 CPU cores Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz. It should not fail with this charge but it does sometimes. This is a screen capture of a typical htop.

capture of a typical htop

When the system fails I can capture its last console message using the IPMI remote control. The last one is this:

last console before crash

With systemd failing to provide these services, the server is unable to work and we can't enter to fix it by ssh, we have to hard reset it. What should we do to prevent this problem?

EDIT: The server has one disk M.2 240GB for the operating system in / and the 12TB disk for /data. The system is ...

Linux tsom02 5.10.0-12-amd64 #1 SMP Debian 5.10.103-1 (2022-03-07) x86_64 GNU/Linux

The M2 is partitioned with only 28GB for /. Maybe that is the reason? Should I use more space for /?

The output of vmstat 5 5 is:

output of vmstat 5 5

cn flag
Does this machine only have 1 disk? It looks like it could be under heavy IO stress. Unfortunately htop doesn't display this information. Please provide the output from 'iostat 5 5' or 'vmstat 5 5' if iostat isn't installed.
us flag
Done wazoox, I hope that helps.
djdomi avatar
za flag
please provide clear text instead of the pictures. it reduces a lot of load tine and makes the read easier, if you still want to use pictures, use an exclamation mark infront of. please provide the output of `df -h`
cn flag
OK, disk doesn't look too heavily loaded. In "last console before crash" there are lots of OOM errors (which can be fatal) and the OOM killer kills python3 processes. Do you know what's the killed application? It's extremely bizarre to have killed python3 processes on a 256GB RAM machine...
cn flag
Also do you have a swap partition?
us flag
Yes, I have a 1Gi swap partition
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.