How can I definitely and permanently increase the open file limit in Ubuntu

Ali EXE

2/15/24, 5:54 AM

I recently used parallel and got this message:

parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n ulimit -Hn)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max

As per message I tried:

ulimit -n `ulimit -Hn`

which by the way I have no idea what does it do.

But I still get that message. Then I saw this and it has nothing new in it. Then I checked this post and added these lines of codes to the /etc/security/limits.conf:

*               soft    nofile          65535
*               hard    nofile          65535

And I again see that warning.

Then I tried to grep -nr ulimit in /etc and in /home directory to see if another limit overrides those values, and I saw no number related to soft/hard nofile.

I have no idea why it does not work.

Is there a definite and permanent way to remove that limit or increase it?

How can I debug to see why it's not applied?

221

0 + 6

bash

parallels

ulimit

matigo

2/15/24, 6:53 AM

You cannot remove the existence of the limit, as all computers have a maximum capacity. That said, this limit is sometimes enforced by the system if the hardware is physically incapable of supporting the configured value. For example, if you were to change the number from 65535 to 1000000, then you would need a remarkable amount of RAM and local storage. So, with this in mind, could you [edit] your question to include your hardware specifications? This will make it much easier to offer some values that your system may not override.

Ali EXE

2/15/24, 7:04 AM

@matigo thank you for mentioning the hardware. I don't want a large number. Even 1024 would suffice. The problem is not hardware here. It's just that whatever I do (even after rebooting) the `parallel` keeps nagging about that limit.

David

2/15/24, 7:37 AM

From your question. which by the way I have no idea what does it do. Why would you ever run a command that you do not know what it does?

Ali EXE

2/15/24, 9:28 AM

@David, I do have a test PC for these tasks.

Doug Smythies

2/15/24, 3:07 PM

How many file handles per job do you need? When the jobs are running what do you get for `lsof | wc -l` and `sudo sysctl fs.file-nr`?

Ali EXE

2/18/24, 1:23 PM

@DougSmythies, basically even 1024 would be enough for me. The problem is that this annoying warning does not go away. And since a lot of our internal scripts are made using `parallel`, any internal command that we run we should see that annoying warning.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How can I definitely and permanently increase the open file limit in Ubuntu

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.