Join Log in
Score:0
ibrahim avatar Server

Gitlab runner fails when running too many job same time

jp flag
ibrahim

I've set some scheduled jobs on gitlab. It is running some parallel jobs every minute and reaches about 500 simultaneous process . After some point runner throws below error and fails.

Running with gitlab-runner 15.5.0 (0d4137b8)
  on XXXX y4s23Bpz
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
ERROR: Job failed (system failure): prepare environment: failed to start process: fork/exec /usr/bin/su: too many open files. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

I've increased open files limit but didn't resolve problem. I couldn't figure out which limit it hits.

~$ ulimit -a 
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515311
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 999999
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 515311
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
289
0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.