Join Log in
Score:2
avatar Server

pthread_create failed (EAGAIN) with most software

vn flag
Twometer

I have a vServer (16GB RAM, 6 vCores) running Ubuntu Server (Linux 4.15.0) with a few docker containers.

A few days ago, I deployed a new container, and it would simply refuse to start with weird native crashdumps from docker-compose, most including these error message:

fork/exec /usr/bin/containerd-shim-runc-v2: resource temporarily unavailable: unknown
runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT:
failed to create new OS thread (have 5 already; errno=11) runtime: may need to increase max user processes (ulimit -u)

After that, this happened with many previously working containers as well. However, I restarted dockerd and all my containers, and the problem seemed to go away.

Today, I wanted to start up my Minecraft server (Forge modded 1.16 on Java 11 without docker) and the problem is back:

[136,281s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

And the weirdest thing so far, it cannot open the TTY device?:

java.io.IOException: Cannot run program "stty": error=11, Resource temporarily unavailable

After these messages, the SSH session just locked up, and a second terminal showed me that I haven't reached a single limit of ulimit -a, and with Minecraft running I still have 9GB of RAM free. htop shows 90 tasks running. I've been working with Linux servers for years now, never seen errors like these - what could be the cause for this?

I know errors like these have been asked before on SF, but not a single one of the answers changed something for me.

988
0 + 7
Michael Hampton avatar
cz flag
Michael Hampton
Please post the output of `virt-what` (install it if it is not present).
0
Reply
vn flag
Twometer
@MichaelHampton It shows `openvz lxc`
0
Reply
Michael Hampton avatar
cz flag
Michael Hampton
Well that's doubly bad news. A container inside a container and you're trying to use Docker? Just stop now and get a real virtual machine from another provider.
0
Reply
vn flag
Twometer
@MichaelHampton Does it mean that my hoster already uses two nested containers? I've actually never had problems up until now
0
Reply
Michael Hampton avatar
cz flag
Michael Hampton
That's what it looks like. Or it could be the new OpenVZ 7, which is based on LXC. Either way you're nesting containers, Docker needs special help to do that, and OpenVZ is very weird with resource allocation anyway; among other things it has a lot of trouble with Java apps. OpenVZ is best avoided by virtually everyone.
1
Reply
vn flag
Twometer
@MichaelHampton okay, I didn't know that... I've been running docker containers on that system for months now and this never happened. So there's basically nothing I can do to fix this except serach for another provider? Another thing I noticed is that since the first docker crash, bash itself sometimes fails to fork() a new process - with the same error message... does that also come from the virtualizer?
0
Reply
vn flag
Twometer
This also seems to just happen when there are already "a lot" of docker containers running (currently I have 18 to be exact)
0
Reply
Score:1
avatar Server
vn flag
Twometer

Finally found the problem: My hoster is limiting the number of threads to 700 in their OpenVZ config.

I can see that in the /proc/user_beancounters. It seems that if numproc goes near 700, these errors occur. Nothing I can do except reduce the number of threads / docker containers.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.