Score:0

Container killed by oom-killer during docker update

in flag

I use docker to host several web services. There are mainly composed with Apache/nginx, PHP and a MySQL database.

Currently, I have 38 containers. If I use docker stats, I see a total of ~4GiB (/ 15.63GiB) used.

free command show the following and confirm the ram used by docker containers :

              total        used        free      shared  buff/cache   available
Mem:           15Gi       4,0Gi       226Mi       447Mi        11Gi        10Gi
Swap:         7,8Gi       622Mi       7,2Gi

During the last maintenance, I noticed an available update from 20.10.6 to 20.10.7 and updated. After the update, ~50% of the containers do not restart with error Exited 137 oom-kill. The containers have the "unless-stopped" restart policy.

During the update the ram available was Ok and clearly show when containers were killed :

RAM space available

In a similar situation (stop/start docker), a reboot, the issue is not present.

I would like to know why these containers were killed with oom-killer ? Should I take care of "free" RAM instead of "available" RAM ? How can I avoid this situation in the future ?

Ginnungagap avatar
gu flag
Look at your RAM consumption when containers start, quite a few services can face a burst of memory during initialization so restating all containers at once might explain the OOMs.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.